Deep Ensembles: A Loss Landscape Perspective
Authors: Stanislav Fort
Clara Huiyi Hu
Balaji Lakshminarayanan
Venue:  (2020)
Keywords: loss landscape
deep ensemble
subspace
tunnel
low loss
connector
weight averaging
dropout
gaussian
connectivity
diversity
function space

Abstract: Deep ensembles have been empirically shown to be a promising approach for improving accuracy, uncertainty  and out-of-distribution robustness of deep learning models. While deep ensembles were theoretically motivated by the bootstrap, non-bootstrap ensembles trained with just random initialization also perform well in practice, which suggests that there could be other explanations for why deep ensembles work well. Bayesian neural networks, which learn distributions over the parameters of the network, are theoretically well-motivated by Bayesian principles, but do not perform as well as deep ensembles in practice, particularly under dataset shift. One possible explanation for this gap between theory and practice is that popular scalable approximate Bayesian methods tend to focus on a single mode, whereas deep ensembles tend to explore diverse modes in function space. We investigate this hypothesis by building on recent work on understanding the loss landscape of neural networks and adding our own exploration to measure the similarity of functions in the space of predictions. Our results show that random initializations explore entirely different modes, while functions along an optimization trajectory or sampled from the subspace thereof cluster within a single mode predictions-wise, while often deviating significantly in the weight space. We demonstrate that while low-loss connectors between modes exist, they are not connected in the space of predictions. Developing the concept of the diversity--accuracy plane, we show that the decorrelation power of random initializations is unmatched by popular subspace sampling methods.

Under review as a conference paper at ICLR 2020
DEEP ENSEMBLES: A Loss LANDSCAPE PERSPECTIVE
Anonymous authors Paper under double-blind review
ABSTRACT
Deep ensembles have been empirically shown to be promising approach for improving accuracy, uncertainty and out-of-distribution robustness of deep learning models. While deep ensembles were theoretically motivated by the bootstrap, non-bootstrap ensembles trained with just random initialization also perform well in practice, which suggests that there could be other explanations for why deep ensembles work well. Bayesian neural networks, which learn distributions over the parameters of the network, are theoretically well-motivated by Bayesian principles, but do not perform well deep ensembles in practice, particularly under dataset shift. One possible explanation for this gap between theory and practice is that popular scalable approximate Bayesian methods tend focus on single mode, whereas deep ensembles tend to explore diverse modes in function space. We investigate this hypothesis by building on recent work on understanding the loss landscape of neural networks and adding our own exploration to measure the similarity of functions in the space of predictions. Our results show that random initializations explore entirely different modes, while functions along an optimization trajectory or sampled from the subspace thereof cluster within a single mode predictions-wice
while often deviating significantly in the weight space. We demonstrate that while low-loss connectors between modes exist, they are not connected in the space of predictions. Developing the concept of the diversity- accuracy plane, we show that the decorrelation power of random initializations is unmatched by popular subspace sampling methods.
INTRODUCTION
Consider typical classification problem, where I,, RD denotes the D-dimensional features and Ya E [I,..., K] denotes the class label. Assume we have a parametric model p(y la, 0) for the conditional distribution where 0 denotes weights and biases of neural network, and p(0) is prior distribution over parameters. The Bayesian posterior over parameters is given by
((0 yn ( p(0) II (V..la,, 0).
Computing the exact posterior distribution over O is computationally expensive (if not impossible) when p(yn (2,,0 0) is deep neural network. A variety of approximations have been developed for Bayesian neural networks, including Laplace approximation (MacKay, 1992), Markov chain Monte Carlo methods (Neal, 1 1996; Welling & Teh, 201 Springenherg et al., 2016), variational Bayesian methods (Graves, 2011: Blundell et al., 2015; Louizos & Welling, 2017; Wen al., 2018) and Monte-Carlo dropout (Gal & Ghahramani, 2016; Srivastava et al., 2014). While computing the posterior challenging, is usually easy to perform maximum- p-ott-rrror (MAP) estimation, which corresponds to mode of the posterior. The MAP solution can be written as the minimizer of the following loss (negative log likelihood + negative log prior):
ÓMAP arg min L(0, (wn, I arg min log ( 1og Kn, 0).
(2)
0
The MAP solution is computationally efficient, but only gives a point estimate and not a distribution parameters. Deep ensembles, proposed by Lakshminarayanan al. (2017), train ensemble

---

Under review as a conference paper at ICLR 2020
of neural networks by initializing at M different values and repeating the minimization multiple times which could lead to M different solutions, if the loss is non non-convex. (Lakshminarayanan et al. (2017) found adversarial training provides additional benefits in some of their experiments, but will ignore adversarial training and focus only on ensembles with random initialization in this paper.)
Given finite training data, many parameter values could equally well explain the observations, and capturing these diverse solutions is crucial for quantifying epistemic uncertainty (Kendall & Gal, 2017). Bayesian neural networks learn a distribution over weights, and a good posterior approximation should be able to learn multi-modal posterior distributions in theory. Deep ensembles were inspired by the bootstrap (Breiman, 1996), which has nice theoretical properties. However, it has been empirically observed by Lakshminarayanan et al. (2017); Lee et al. (2015) that training individual networks with just random initialization is sufficient in practice and using the bootstrap even hurts performance in some cases (e.g. for small ensemble sizes). Furthermore, Ovadia et al. (2019) and Gustafsson et al. (2019) independently benchmarked existing methods for uncertainty quantification on variety of datasets and architectures, and observed that ensembles tend to outperform approximate Bayesian neural networks in terms of both accuracy and uncertainty, particularly under dataset shift.
These empirical observations raise an important question: Why do ensem- bles trained with just random initial- ization work so well in practice? One possible hypothesis is that ensembles tend to sample from different modes! in function space, whereas variational Bayesian methods (which minimize (g [z)) yo ( 1) might fail to explore multiple modes even though they are effective at captur- ing uncertainty within a single mode. See Figure for a cartoon illustration. Note that while the MAP solution is a local minima for the training loss by definition, it may not necessarily be a local minima for the validation loss.
Ensocandes indeutiffy diffecent modes but yeeen locad mnce uncertaiity and migle not pins the bost point from each mock
Vactactonal methocls expetie pocdl moveraliry aroud unde
Validation
Space of solutions
Teainiog
Figure 1: Cartoon illustration of the hypothesis. z-axis in- dicates parameter values and y-axis plots the negative loss -I, (In, Yo ) on train and validation data.
Recent work on understanding loss landscapes (Fort& & Jastrzebski, 2019; Draxler et al., 2018; Garipov al., 2018) allows us to investigate this hypothesis. Note that prior work loss landscapes has focused on cone-coonntt and low-loss tunnels, but has not explicitly focused on how diverse the functions from different modes are, beyond an initial exploration in Fort & Jastrzebski (2019). Our findings show that:
The functions sampled along a single training trajectory or subspace thereof (e.g. diagonal Gaussian, low-rank Gaussian and Dropout subspaces) tend to be very similar in predictions (while potential far away in the weight space), whereas functions sampled from different randomly initialized trajectories tend to be very diverse.
Solution modes are connected in the loss landscape but they are distinct in the space of predictions. Low-loss tunnels create functions with near-identical low values of loss along the path, however these functions tend to be very different in function space, changing significantly in the middle of the tunnel.
2 BACKGROUND
The loss landscape of neural networks (also called the objective landscape) - the space of weights and biases that the network navigates is typically very high dimensional function and therefore could potentially be very complicated. However, many empirical results show interesting properties of the loss surface. Goodfellow & Vinyals (2014) observed that the loss along linear path from an initialization to the corresponding optimum is monotonically decreasing, encountering no significant
We use the term mode to refer to unique functions fo(x). Due to weight space symmetries, different parameters could correspond to the same function, i.e. fo, (x) for (az) even though 0 a 0z. but we ignore this aspect and leave it future work.
2

---

Under review as a conference paper at ICLR 2020
obstacles along the way. Li et al. (2018) demonstrated that constraining optimization random, low-dimensional hyperplane in the weight space leads to results comparable to full-space optimization, provided that the dimension exceeds modest threshold. This was geometrically understood and extended in (Fort & Scherlis, 2019). Garipov et al. (2018), 8); Draxler et al. (2018) demonstrate that while a linear path between two independent optima hits high loss area in the middle, there in fact exist continuous, low-loss paths connecting any pair of optima. These observations are unified into a single phenomenological model in (Fort & Jastrzebski, 2019). While independent, low-loss optima in the loss landscape are connected, Fort & Jastrzebski (2019) provide an early indication that in fact they represent very different functions in terms of their predictions. Therefore the connectivity cannot be due to trivial symmetries of the network which would keep the input-output mapping intact.
3 VISUALIZING FUNCTION SIMILARITY ACROSS INITIALIZATIONS
We train convolutional neural networks on the CIFAR-10 (Krizhevsky, 2009) dataset:
SmalICNN: channels [16,32,32] for 10 epochs which achieves 64% test accuracy.
MediumCNN: channels [64, 128, 128,256,256) for 20 epochs which achieves 70% test accuracy.
ResNer20v1: for 200 epochs which achieves 90% test accuracy.
We use the Adam optimizer (Kingma & Ba, 2015) for training and to make sure the effects we observe are general, we validate that our results hold for vanilla stochastic gradient descent (SGD) as well, which we do not show in this paper. We use batch size 128 and dropout 0.03 for training SmallCNN and MediumCNN, To generate weight space and prediction space similarity results, we use a constant learning rate of 1.6 x 10-3 unless specified otherwise. We do not any data augmentation with those two architectures. For ResNet20vl,, we use the data augmentation and learning rate schedule used in Keras examples The overall trends are consistent across all architectures, datasets, and other hyperparameter and non-linearity choices we explored.
3.1 SIMILARITY OF FUNCTIONS WITHIN AND ACROSS TRAJECTORIES
First, we compute the similarity between different checkpoints along a single trajectory. We plot the cosine similarity in weight space in Figure 2(a) and the disagreement in function space, defined the fraction of points the checkpoints disagree on, in Figure 2(b). We observe that the checkpoints along trajectory are largely similar both in the weight space and the function space. Next, we evaluate how diverse the final solutions from different random initializations are. The functions from different initialization are different, demonstrated by the similarity plots in Figure 3. Comparing this with Figures 2(a) and 2(b). we see that functions within a single trajectory exhibit higher similarity and functions across different trajectories exhibit much lower similarity.
Next, we take the predictions from different checkpoints along the individual training trajectories from multiple initializations and compute a t-SNE plot (Maaten Hinton, 2008) to visualize their similarity in function space. More precisely, we take the softmax output for a set of points, flatten the vector and use as the input to the t-SNE plot. Figure 2(c) shows that the functions explored by different trajectories (denoted by circles with different colors) are far away, while functions explored within a single trajectory (circles with the same color) tend be much more similar.
3.2 SIMILARITY OF FUNCTIONS ACROSS SUBSPACES FROM EACH TRAJECTORY
In addition to the checkpoints along a trajectory, also construct subspaces based on each individual trajectory. Scalable Bayesian methods typically compute statistics based on the weights along a trajectory, hence visualizing the diversity of functions between the subspace helps understand the difference between Bayesian neural networks and ensembles. We use a representative set of four subspace sampling methods: a random subspace, a Monte Carlo dropout, a diagonal Gaussian approximation, and a low-rank covariance matrix Gaussian approximation. In the descriptions of the methods, let wh be the current weight-space position (the weights and biases of our trained neural net) around which we will construct the subspace.
Phttps://keraa 10/ wttpxxaalee//eeeeeeeeeeeeeeeeeee far resnet/
3

---

Under review as a conference paper at ICLR 2020
1.30
21-
1.8
123
C33
0.2
20
Chockpo
Crockportt
(a) Cosine similarity (weight space) (b) Disagreement (prediction space) (c) t-SNE of predictions
Results using SimpleCNN on CIFAR-10. Left plot: Cosine similarity between checkpoints measure weight space alignment along optimization trajectory. Middle plot: The fraction of labels on which the predictions from different checkpoints disagree. Right plot: t-SNE plot of predictions from checkpoints corresponding 3 different randomly initialized trajectories (in different colors).
Figure 2:
20
.015
(a) Results using SimpleCNN
(b) Results using ResNet
Figure 3: Results on CIFAR-10 using two different architectures. For each of these architectures, the
left subplot shows the cosine similarity between different solutions in weight space, and the right subplot shows the fraction labels on which the predictions from different solutions disagree.
Random subspace sampling: We start optimized solution wh and choose random direction the weight space. We step in that direction by choosing different values and looking at predictions configurations Wo tô. We do this for many random directions é.
Monte Carlo dropout subspace: We start at optimized solution W. and apply dropout with randomly chosen Pkeep it. We do this many times, each time choosing random Pkeep and look at predictions at dropout (wop).
Diagonal Gaussian subspace: We start at an optimized solution wo and look at the most recent iterations of training proceeding it. For each trainable parameter W), we calculate its mean mean; and standard deviation std(w:). To sample solutions from the subspace, independently as w. ~ N (mean,, std,). We repeat this many times and obtain predictions in each. This corresponds sampling from normal distribution with diagonal covariance matrix.
draw each parameter
Low-rank Gaussian subspace: We start at an optimized solution wp and look at the most recent iterations of training proceeding it. For each trainable parameter Wi calculate its mean mean,- For rank-k approximation, we calculate top A principal components of the weight vectors in the most recent iterations of training {Ø; paarans kk We sample from k-dimensional normal distribution and obtain the weight configurations as ő mean + Σ, NO p
Figure 4 shows that functions sampled from subspace (denoted by colored squares) corresponding particular initialization, are much more similar to each other. While some subspaces are more diverse, they still do not overlap with functions from another randomly initialized trajectory.
Diversity versus Accuracy plots To illustrate the difference another fashion, we sample functions from single subspace and plot diversity (as measured by disagreement between predictions) versus accuracy in Figure 5. Comparing these subspace points (colored dots) to the baseline optima (green star) and the optima from different random initializations (denoted by red stars), observe that random initializations are much more effective at sampling diverse and accurate solutions, than subspace based methods constructed out of single trajectory.
4

---

Under review as a conference paper at ICLR 2020
Figure 4: Results using SimpleCNN CIFAR-10: t-SNE plots of validation set predictions for each trajectory along with four different subspace generation methods (showed by squares), in addition to 3 independently initialized and trained runs (different colors). As visible in the plot, the subspace- sampled functions stay in the prediction-space neighborhood of the run around which they were constructed, demonstrating that truly different functions are not sampled.
Simele CNN CIFAR-10
Medium CIFAR.10
Bestaratvt CIPAR-10
Uppar
Upper
sutspaace
snnt
independert optins
independert optins
Independert optins
We
baseline optimum
MAe basdlng optimum
Mhx baseline optimum
61
24
08
Figure 5: Diversity versus accuracy plots for 3 models trained on CIFAR- 10: SmallCNN, Medium- CNN and ResNet20vl. The clear separation between the subspace sampling populations (for different subspace sampling methods) and the population of independently initialized and optimized solutions (red stars) visible. The limiting curves correspond solution generated by perturbing the reference solution's predictions (bottom curve) and completely random predictions at given accuracy (upper curve).
The diversity score used above quantifies the difference of two functions, by measuring fraction of points on which their predictions differ. We chose this approach due to its simplicity; one could also compute the KL-divergence other distances between the output probability distributions Let dait denote the fraction of predictions on which the two functions differ. It is O when the two functions make identical class predictions, and 1 when they differ on every single example. To account for the fact that the lower the accuracy of function, the higher its potential dait due to the possibility of the wrong answers being random and uncorrelated between the two functions, we normalize this by (1 - a), where a is the accuracy.
For a reference function f* of accuracy a" and a function f of accuracy a whose predictions obtained by randomly perturbing the predictions of f*, the expected fractional difference is is the number of classes. If the function of accuracy were entirely independent of fr, then the expected fractional difference would be dai = (1 a )a + (1 a)a" + (1- a²) d1) (1- a) 2)/(C -1)- Those two limiting behaviours the function being derived from f by perturbation, and and f being completely independent form the two dashed lines in Figure 5. We refer to Appendix for further details on the limiting curves. The diversity reached is not as high as the theoretical optimum even for the independently initialized and optimized solutions, which provides scope for future work.
daitt = (C - 1)(a" - a)/(a"C -CC-) where
3.3 IDENTICAL LOSS DOES NOT IMPLY IDENTICAL FUNCTIONS IN PREDICTION SPACE
Figure 6 shows the radial loss landscape (train as well as the validation set) along the directions of two different optima. The left subplot shows that different trajectories achieve similar values of the loss, and the right subplot shows the similarity of these functions to their respective optima (in particular the fraction of labels predicted on which they differ divided by their error rate). While the loss values from different optima are similar, the functions are different, which confirms that random initialization leads to different modes in function space.
5

---

Under review as a conference paper at ICLR 2020
475
340
Figure 6: Results using MediummNN on CIFAR-1O: Radial loss landscape cut between the origin and Two independent optima and the predictions of models on the same plane.
We construct a low-loss tunnel between different optima using the procedure proposed by Fort & Jastrzebski (2019). which is simplification of the procedures proposed in Garipov al. (2018) and Draxler et al. (2018) 8). As shown in Figure 7(a), we start at the linear interpolation point (denoted by the black line) and reach the closest point on the manifold by minimizing the training loss. The minima of the training loss are denoted by the yellow line in the manifolds. Figure 7(b) confirms that the tunnel is indeed low-loss.
In order to visualize the 2-dimensional cut through the loss landscape and the the associated pre- dictions on along a curved low-loss path, we divide the path into linear segments, and compute the triangle given by this segment
loss and prediction similarities weight space on the other. We perform this operation on each of the linear segments from which the low-loss path is constructed, and place them next each other for visualization. Figure 8 visualizes the loss along the manifold, as well as the similarity to the original optima. Note that the regions between radial yellow lines consist of segments, and stitch these segments together in Figure 8. The accuracy plots show that as we traverse along the low-loss tunnel, the accuracy remains fairly constant expected. However, the prediction similarity plot shows that the low-loss tunnel does not correspond to similar solutions in function space. What it shows is that while the modes are connected terms of accuracy/loss, their functional forms remain distinct and they do not collapse into a single mode.
one side and the origin of the
Monstold
Munifakd
Plano
Oytlamn
Oatimma
0.20
0.5
0.10
0.65
0.1
0.00
Interpolation
Interpolation
(a) Cartoon illustration
(b) Low-loss tunnel
Figure 7: Left: Cartoon illustration showing linear connector (black) along with the optimized connector which lies the manifold of low loss solutions. Righr: The loss and accuracy in between two independent optima on a linear path and an optimized path in the weight space.
4 EVALUATING THE RELATIVE EFFECTS OF ENSEMBLING VERSUS SUBSPACE METHODS
Our observations in the previous section suggest that subspace- based methods and ensembling should provide complementary benefits in terms of uncertainty and accuracy. To test this, we evaluate the performance of the following four variants using SmalICNN on CIFAR-10:
Baseline: optimum at the end of a single training trajectory.
Subspace sampling: average predictions over the solutions sampled from subspace.
Ensemble: train baseline multiple times with random initialization and average the predictions.
Ensemble + Subspace sampling: train multiple times with random initialization, use subspace sampling within each trajectory.

---

Under review as a conference paper at ICLR 2020
Training accurssy
Valication set sccurcy
0.0
0.90
099
0.34
034
-03
optimum 0-30
origin
-1.
int
suiches
-10
-15
0.5
3.0
Tonart oommn
Sipilarity to optimum
Similarity optimurp
0.83
0.0
Tenunbb o0:mmmm
0.54
0.55
0.5
optimum setches
optimium
-.4
155
3.5
2.5
Figure 8: Results using MeNiumCNN on CIFAR-10: Radial loss landscape cut between the origin and two independent optima along an optimize low-loss connector and predictions similarity along the same planes.
Figures 9a) and 9(b) show the results for low rank Gaussian subspace and diagonal Gaussian subspace respectively. The results validate our hypothesis as (i) subspace sampling and ensembling provide complementary benefits, and (ii) the relative benefits of ensembling are higher averages predictions over more diverse solutions.
Low rank Gaussian: Accuracy and Brier
(b) Diagonal Gaussian: Accuracy and Brier
Figure 9: Results on CIFAR-10 showing the complementary benefits of ensemble and subspace methods, as well as the effect of ensemble size.
Weight averaging within subspace One could the mean and diagonal/low variance to approximate each mode of the posterior, however that increases the number of parameters required for each mode. Using just the mean weight for each mode would increase the number of parameters. Izmailov et al. (2018) proposed stochastic weight averaging (SWA) for better generalization. One could also compute an (exponential moving) average the weights along the trajectory, inspired by Folyak-Rupper averaging in convex optimization, (see also (Mandt et al., 2017) for a Bayesian view on iterate averaging). As weight averaging has been already studied by Izmailov et al. (2018), we do not discuss it in detail. Figure S1 provides illustration of why these strategies might help with generalization. We use weight averaging (WA) on the last few epochs which corresponds to using the mean of the subspace within each mode. Figure 10(a) shows that weight averaging achieves better performance within each mode, and ensemble WA performs as well as ensemble + subspace combination methods, without any additional parameter overhead.
7

---

Under review as a conference paper at ICLR 2020
By Bud Bre Ercand
(a) Accuracy & Brier: Weight Averaging vs. Ensemble (h) Results on CIFAR-10-C: Accuracy & Brier versus Corruption Intensity
Figure 10: Results on CIFAR-10 using SimpleCNN: clean test and CIFAR-10-C corrupted test set.
Figure 10(b) shows accuracy and Brier score on CIFAR-10, both on the usual test set (corresponding the intensity column) as well as on the CIFAR- CEFAR-10-C benchmark proposed (Hendrycks & Dietterich, 2019) which contains corrupted versions of CIFAR-10 with varying intensity values (1-5), making useful to verify calibration under dataset shift (Ovadia et al., 2019). We see that ensembling and weeight-averaging provide complementary benefits. WA improves over the vanilla baseline, but combining WA with ensembling over multiple random initializations improves performance further. Figure reports accuracy and Brier size. Under dataset shift, is particular important to have diverse functions to avoid overconfident predictions (as averaging over similar functions would not reduce overconfidence).
the usual CIFAR-10 0 test
function of ensemble
4.1 RESULTS ON IMAGENET
To illustrate the effect on another challenging dataset, we repeat these experiments on ImageNet (Deng et al., 2009) using the same ResNet20V1 architecture. Due to computational constraints, we focus mainly on the experiment decoupling the effect of weight averaging ensembling. Figure 11(a) shows the complementary effects of ensembling and weight averaging; Figure 11(b) shows results on (subset of) ImageNet-C demonstrating that these trends similar those observed on CIFAR-10.
indaccar
(a) Accuracy & Brier: Weight Averaging vs. Ensemble (b) Results ImageNet-C: Accuracy & Brier versus Corruption Intensity
Figure 11: Results using ResNet on ImageNet: clean test and ImageNet-C corrppee test set.
DISCUSSION
Our results show that trajectories of randomly initialized neural networks explore different modes in function space, which explains why deep ensembles with random initializations help. They are essentially orthogonal to each other in the space of weights and very diverse in terms of their predictions. While these modes can be connected via optimized low-loss paths between them, we demonstrate that they correspond to distinct functions in terms of their predictions. Therefore the connectivity in the loss landscape does not imply connectivity in the space of functions.
Subspace sampling methods such as weight averaging. Monte Carlo dropout, and various versions of local Gaussian approximations, sample functions that might lie relatively far from the starting point in the weight space, however, they remain in the vicinity of their starting point in terms of predictions, giving rise to an insufficiently diverse set functions. Using the concept of the diversity- accuracy plane, demonstrate empirically that these subspace sampling methods never reach the combination of diversity and accuracy that independently trained models do, limiting their usefulness for ensembling.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. ICML, 2015.
Leo Breiman. Bagging predictors. Machine learning, 24(2):123-140, 1996.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In Computer Vision and Putlern Recognition, 2009.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. arXiv preprint arXiv: mat::0803000885 2018.
Stanislav Fort and Stanislaw Jastrzebski. Large scale structure of neural network loss landscapes. arXiv preprint arXiv: 1906:0090 2019.
Stanislav Fort and Adam Scherlis. The Goldilocks zone: Towards better understanding of neural
network loss landscapes. In AAAI, 2019.
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In ICML, 2016.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of DNNs. In NewrlPS, 2018.
Ian J. Goodfellow and Oriol Vinyals. Qualitatively characterizing neural network optimization problems. CoRR, abs/ 0alds/1412.5544,, 2014.
Alex Graves. Practical variational inference for neural networks. In NeurlPS, 2011.
Fredrik K Gustafsson, Martin Danelljan, and Thomas B Schön. Evaluating scalable Bayesian deep learning methods for robust computer vision. arXiv preprint arXiv:1906.0166 096220006000,, 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Av- eraging weights leads to wider optima and better generalization. arXiv preprint 2018.
Alex Kendall and Yarin Gal. What uncertainties do we need in Bayesian deep learning for computer vision? In NeurIPS, 2017.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Balaji Lakshminarayaman, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NeurlPS, 2017.
Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why M heads are better than one: Training a diverse ensemble of deep networks. arXiv preprint arXiv:1511.06314. 2015.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. In ICLR, 2018.
Christos Louizos and Max Welling. Multiplicative Normalizing Flows for Variational Bayesian Neural Networks, In ICML, 2017.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 97N0v)25579-600 2008
David JC MacKay. Bayesian methods for adaptive models. PhD thesis, California Institute of Technology, 1992.

---

Under review as a conference paper at ICLR 2020
Stephan Mandt, Matthew D Hoffman, and David M Blei. Stochastic gradient descent approximate Bayesian inference. The Journal of Machine Learning Research, (8(1):487344907, 2017.
Radford M. Neal. Bayesian Learning, for Neural Networks. Springer- New York, Inc., 1996.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D Sculley, Sebastian Nowozin, Joshua V Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. arXiv preprint arXiv: 02530, 2019.
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian optimization with robust Bayesian neural networks. In NeurIPS, 2016.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way prevent neural networks from overfitting JMLR, 2014.
Max Welling and Yee Whye Teh. Bayesian Learning via Stochastic Gradient Langevin Dynamics. In ICML, 2011.
Yeming Wen, Paul Vicol, Jimmy Ba, Dustin Tran, and Roger Grosse. Flipout: Efficient pseudo- independent weight perturbations on mini-batches. arXiv preprint 183:::880.004386, 2018.
10

---

Under review as a conference paper at ICLR 2020
A VISUALIZING THE LOSS LANDSCAPE ALONG ORIGINAL DIRECTIONS AND WA DIRECTIONS
Figure S1 shows the loss landscape (train well the validation set) and the effect of WA.
Training set accuracy
0.72
Validation accuracy
0.72
1.0
2.00
0.50
Towarts opcmum
0.-
2.40
0.40
-0.3
-1.0-
origin
3.16
optimum a=30
0.36
init
-1.0
-05
56
05
1.0
1.5
Towards optimums
Training accuracy
Validation set pccuracy
0.72
0.72
3.5
0.66
Towaros meae weights of aptmmmm
0.50
0.54
0.54
30-
-.42
-042
optimum <c-(26, 27, 28, 29, 301
optimum
30]
-10
origin
Init
Init
1.0
2.50
-1.0
0.0
05
1.5
1.5
0.30
Towares moon woightz of opcmum
Figure S1: Loss landscape versus generalization: weights are typically initialized close to 0 and increase radially through the course of training. Top row: we pick two optima from different trajectories the axes, and plot loss surface. Looking and axes, we observe that while wide range of radii achieve low loss on training set, the range of optimal radius values is narrower on validation Bottom axes. A wider range of radius values generalize better along the WA directions, which confirms the findings of Izmailov al. (2018).
average weights within each trajectory using WA and use them as
B ADDITIONAL ABLATION EXPERIMENTS
B.1 EFFECT OF RANDOMNESS: RANDOM INITIALIZATION VERSUS RANDOM SHUFFLING
Random seed affects both initial parameter values as well the order of shuffling of data points. We run experiments to decouple the effect of random initialization and shuffling; Figure S2 shows shows the results. We observe that both of them provide complementary sources of randomness, with random initialization being the dominant of the two. As expected, random mini-batch shuffling adds more randomness at higher learning rates due to gradient noise.
C ADDITIONAL DIVERSITY ACCURACY RESULTS ON CIFAR- 100
We run additional experiments comparing the diversity of solutions found vs their test accuracy on CIFAR-100. CIFAR- 100 is intermediate step between CIFAR-10 and ImageNet, and overall
11

---

Under review as a conference paper at ICLR 2020
0.40
0.30
Fraction of labels different
0.25
Random inits, random batches
Random inits, fixed batchès
Fixed init, raniom batches
Fixed init, fixed batches
0.15
0.10
0.00
x 10
8 10
16 x 10
Learning rate
Figure S2: The effect of random initializations and random training batches the diversity of predictions.
much more challenging to learn than CIFAR-10 10. Our additional results are presented in Figure S3. Solutions obtained by subspace sampling methods described in Section 4 have worse trade off between prediction diversity (needed for ensembling) and accuracy, compared to independently initialized and trained optima. This consistent with our results on CIFAR-10 Figure 5.
ResNet20y1 CIFAR-100
Predictio diversity accurscy! changed
2.0
0.8
Upper limit
Lower
0.6
random subspace
dropout subspace
simple gaussian
0.2
gaussian
independent optima
baseline optimum
0.1
accuracy
Figure S3: Diversity versus accuracy plots for a ResNet20v] trained on CIFAR-100.
12

---

Under review as a conference paper at ICLR 2020
D DERIVING THE UPPER AND LOWER LIMIT CURVES IN THE DIVERSITY-ACCURACY PLOTS
In Figures 5 and S3 bound our empirical results by theoretically derived curves, limiting the expected trade off between diversity and accuracy in the best and worst case scenarios. The resulting functions are presented in the main text in Section 3.2. We will show the detailed derivations here.
Given a C-class classification problem and reference solution with accuracy a', we would like to obtain a function daa (a) which gives the fraction of labels on which another solution disagrees with the reference solution as function of its accuracy.
D.1 UNCORRELATED PREDICTIONS THE BEST CASE
The best case scenario is when the predicted labels are uncorrelated with the reference solution's labels. On particular example, the probability that the reference solution got it correctly is a", and the probability that our solution got it correctly is a. On those examples, the predictions do not differ since they both have be equal the ground truth label. The probability that the reference solution correct on an example while our solution is wrong is a a²(1 - a). The probability that the reference solution is wrong on an example while our solution correct (1 a )a. On the examples where both solutions are wrong (probability (1 d1 - a)) there are two cases; a) the two solutions agree (an additional factor of LC(C- - 1)) or b) disagree (an additional factor of (C 2)/(C' 1)). Only the case b) contributes to the fraction of labels on which they disagree. Hence we end up with
C-2
daia (a; a, C) = (1 a da)a + (1 - a)a" + (1 - a²)(1
(3)
C-1
D.2 CORRELATED PREDICTIONS THE WORST CASE
The other extreme case is when the predictions of our new solution are just the predictions of the reference solution perturbed by perturbations of different strength. Then, the solutions retain great amount of correlation.
Let the probability of label changing be p. We will consider 4 cases: a) the label of the correctly
classified image does not flip (probability a=(1 (1 - p)), b) it flips (probability a" a²p), c) an incorrectly a² (1 p)), and d) it flips (probability (1 a p))
labelled image does not flip (probability
The resulting accuracy a(p) obtains a contribution a' a"(1 - p) from case a) and with probability 1/(C 1) contribution (1 a*) ") from d). Therefore a(p) a" (1 p) + p(1 a //C(C 1).
Inverting this relationship, we get p(a)= = (C - 1)(a* a)/(Ca" - 1). The fraction of labels on which the solutions disagree is simply p by our definition of p. and therefore
(a a", CC (C 1)((a 1a)
Ca"
13


Reviews:
[{"review_id": "rkl8KjQ6YH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is trying to answer the question why ensembles of deep neural networks trained with random initialization work so well in practice in improving accuracy. Their proposed hypothesis is that networks trained from different initializations, although all converge to a low-loss/high accuracy optimum, explore different modes in function space and therefore provide more diversity. To experimentally support their hypothesis, first they show that functions along a single training trajectory are similar, however trajectories starting from different initializations may significantly differ. The difference in function space is based on the fraction of points on which the two functions disagree in terms of their prediction. Second, they use different subspace sampling methods around a single optimum and demonstrate that they are significantly less diverse (low disagreement between predictions) than sampling from independent optima through diversity vs accuracy plots. Moreover, they comment on the recent observation that local optima are connected by low-loss tunnels. They experimentally show that even though low-loss/high accuracy path exists between local optima, these tunnels do not correspond to similar solutions in function space, further supporting the multi-mode hypothesis. The authors compare the relative benefit of subspace sampling, weight averaging and ensembling on accuracy and interpret their findings in terms of the hypothesis. \n\nOverall, the paper is very well written and provides interesting insights into the multi-modal structure of deep neural network loss landscapes. Even though the hypothesis of the paper is not entirely new and has been touched upon in Fort & Jastrzebski (2019), this paper contributes to the field by providing thorough experimental support and clear exposition of the idea. Therefore, I would accept this paper if the authors provided additional experimental results on a different dataset.\n\nThe paper mentions that the trends are consistent across all datasets the authors have explored. However, they only provide results on CIFAR-10 (and a limited set of experiments on ImageNet). Since the contribution of the paper heavily relies on providing experimental verification, it would be important to include at least the diversity vs. accuracy plot for the other datasets they have explored to demonstrate that this phenomenon is not specific to CIFAR-10. \n\nAdditionally, I would like to add a couple of comments on the paper that are not part of my decision, but could potentially improve the paper. \n-The diversity score introduced in the paper is simple and intuitive, however it would be interesting to see whether the results hold across different notions of function space disagreement. \n\n-It is mentioned in the paper that data augmentation has been used for training the ResNet20 architecture. Would the results change significantly without data augmentation, as it adds another source of randomness to the training procedure.\n\n-Some comments on the figures: in Figure 3 it is very difficult to discern any difference between different shades of red (disagreement values), and in this form the plots are not too informative. Maybe rescaling or a different way of presentation would help. Interpreting Figure 7/a is a bit difficult, probably a 3D plot would be useful to explain the different line sections.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1l0JeopFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The contribution of the paper is the following two findings: 1. Despite the fact that local minima are connected in the loss landscape the functions corresponding to the points on the curve are significantly distinct. 2. The points along the training trajectory correspond to similar functions. \n\nOriginality and novelty. Both findings do not seem quite new. The first conclusion can be mostly derived from Figure 2 right [1]. Moreover, the difference between functions on the curve in terms of predictions is the main motivation of Fast Geometric Ensembling. The second conclusion is also not quite new and there were several approaches to overcome it e.g. SWA [2]. I appreciate that the authors did a much broader investigation of this phenomena than it was done in previous works. Another drawback is lack of practical implications. It is known that ensembling based on dropout is worse than independent networks, but the main advantage of this and similar approaches is memory efficiency. \n\nThe clarity. The paper is well written, contains all necessary references and is easy to follow. The provided experimental results and supporting plots are also clear and contain the necessary description. The only part that I found a bit confusing is radial plots. I would recommend the authors to add more rigorous description of how they constructed these plots to increase clarity of the paper. Can the authors please also clarify how they derived formulas for the expected fractional difference for f^* and f functions in the section 3.2? \n\nOverall, it is an interesting paper, but the findings are not quite new.\n[1]  Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of DNNs. InNeurIPS, 2018\n[2] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Av-eraging weights leads to wider optima and better generalization.arXiv preprint arXiv:1803.05407,2018\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BJxk8KaptB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper analyzes ensembling methods in deep learning from the perspective of the loss landscapes. The authors empirically show that popular methods for learning Bayesian neural networks produce samples with limited diversity in the function space compared to modes of the loss found using different random initializations. The paper also considers the low-loss paths connecting independent local optima in the weight-space. The analysis shows that while the values of the loss and accuracy are nearly constant along the paths, the models corresponding to different points on a path define different functions with diverse predictions. The paper also demonstrates the complementary benefits of using subspace sampling/weight averaging in combination with deep ensembles and shows that relative benefits of deep ensembles are higher. \n\nThe paper is well-written. The experiments are described well and the results are presented clearly in highly-detailed and visually-appealing figures. There are occasional statements which are not formulated rigorously enough (see comments below).\n\nThe paper presents a thorough experimental study of different ensemble types, their performance, and function space diversity of individual members of an ensemble. In my view, the strongest contribution of the paper is the analysis of the diversity of the predictions for different sampling procedures in comparison to deep ensembles. However, the novelty and the significance of the other contributions are limited (see comments below). Therefore, I consider the paper to be below the acceptance threshold.   \n\nComments and questions to authors:\n1) The practical aspects of different ensembling techniques are not discussed in the paper. While it is known that deep ensembles generally demonstrate stronger performance [1], there is a trade-off between the ensemble performance and training time/memory consumption. The considered alternative ensembling procedures can be favorable in specialized settings (e.g. limited training time and/or memory).\n\n2) It remains unclear to me what new insights does the analysis of the low-loss connectors provide? It is expected (and in fact can be shown analytically) that if the two modes define different functions then intermediate points on a continuous path define functions which are different from those defined by the end-points of the path. This result was also analyzed before from the perspective of the performance of ensembles formed by the intermediate points on the connecting paths (see Fig. 2 right in [2]).\nMoreover, I would encourage authors to reformulate the statements on the connectivity in the function space such as: \n-- “We demonstrate that while low-loss connectors between modes exist, they are not connected in the space of predictions.”  (Abstract)\n-- “the connectivity in the loss landscape does not imply connectivity in the space of functions” (Discussion)\nIn my opinion, these claims are somewhat misleading. What does it mean that the modes are disconnected in the function space? Neural networks define continuous functions (w.r.t to both the inputs and the weights), and a connector is continuous path in the weight space which continuously connects the modes in the function space (i.e. a path defines a homotopy between two functions). It is true that two modes correspond to two different functions. However, it is unclear in which sense these functions can be considered to be disconnected. \n\n[1] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NeurIPS, 2017.\n\n[2] Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of DNNs. In NeurIPS, 2018.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Paper https://arxiv.org/abs/1802.10026 (Garipov et. al, NeurIPS 2018) shows that one can find curves between two independently trained solutions along which the loss is relatively constant. The authors of this ICLR submission claim as a key contribution that they show the weights along the path correspond to different models that make different predictions ("Note that prior work on loss landscapes has focused on mode-connectivity and low-loss tunnels, but has not explicitly focused on how diverse the functions from different modes are, beyond an initial exploration in Fort & Jastrzebski (2019)"). Much of the disagreement between two of the reviewers and the authors is whether this point had already been shown in 1802.10026.

It is in fact very clear that 1802.10026 shows that different points on the curve correspond to diverse functions. Figure 2 (right) of this paper shows the test error of an _ensemble_ of predictions made by the network for the parameters at one end of the curve, and the network described by \phi_\theta(t) at some point t along the curve: since the error goes down and changes significantly as t varies, the functions corresponding to different parameter settings along these curves must be diverse. This functional diversity is also made explicit multiple times in 1802.10026, which clearly says that this result shows that the curves contain meaningfully different representations.

In response to R3, the authors incorrectly claim that  "Figure 2 in Garipov et al. only plots loss and accuracy, and does not measure function space similarity, between different initializations, or along the tunnel at all. Just by looking at accuracy and loss values, there is no way to infer how similar the predictions of the two functions are." But Figure 2 (right) is actually showing the test error of an average of predictions of networks with parameters at different points along the curve, how it changes as one moves along the curve, and the improved accuracy of the ensemble over using one of the endpoints. If the functions associated with different parameters along the curve were the same, averaging their predictions would not help performance. 

Moreover, Figure 6 (bottom left, dashed lines) in the appendix of 1802.10026 shows the improvement in performance in ensembling points along the curve over ensembling independently trained networks. Section A6 (Appendix) also describes ensembling along the curve in some detail, with several quantitative results. There is no sense in ensembling models along the curve if they were the same model.

These results unequivocally demonstrate that the points on the curve have functional diversity, and this connection is made explicit multiple times in 1802.10026 with the claim of meaningfully different representations: “This result also demonstrates that these curves do not exist only due to degenerate parametrizations of the network (such as rescaling on either side of a ReLU); instead, points along the curve correspond to meaningfully different representations of the data that can be ensembled for improved performance.”  Additionally, other published work has built on this observation, such as 1907.07504 (UAI 2019), which performs Bayesian model averaging over the mode connecting subspace, relying on diversity of functions in this space; that work also visualizes the different functions arising in this space. 

It is incorrect to attribute these findings to Fort & Jastrzebski (2019) or the current submission.  It is a positive contribution to build on prior work, but what is prior work and what is new should be accurately characterized, and currently is not, even after the discussion phase where multiple reviewers raised the same concern. Reviewers appreciated the broader investigation of diversity and its effect on ensembling, and the more detailed study regarding connecting curves. In addition to the concerns about inaccurate claims regarding prior work and novelty (which included aspects of the mode connectivity work but also other works), several reviewers also felt that the time-accuracy trade-offs of deep ensembles relative to standard approaches were not clearly presented, and comparisons were lacking. It would be simple and informative to do an experiment showing a runtime-accuracy trade-off curve for deep ensembles alongside FGE and various Bayesian deep learning methods and mc-dropout. It's also possible to use for example parallel MCMC chains to explore multiple quite different modes like deep ensembles but for Bayesian deep learning. For the paper to be accepted, it would need significant revisions, correcting the accuracy of claims, and providing such experiments.
Decision: Reject