Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper analyzes ensembling methods in deep learning from the perspective of the loss landscapes. The authors empirically show that popular methods for learning Bayesian neural networks produce samples with limited diversity in the function space compared to modes of the loss found using different random initializations. The paper also considers the low-loss paths connecting independent local optima in the weight-space. The analysis shows that while the values of the loss and accuracy are nearly constant along the paths, the models corresponding to different points on a path define different functions with diverse predictions. The paper also demonstrates the complementary benefits of using subspace sampling/weight averaging in combination with deep ensembles and shows that relative benefits of deep ensembles are higher. 

The paper is well-written. The experiments are described well and the results are presented clearly in highly-detailed and visually-appealing figures. There are occasional statements which are not formulated rigorously enough (see comments below).

The paper presents a thorough experimental study of different ensemble types, their performance, and function space diversity of individual members of an ensemble. In my view, the strongest contribution of the paper is the analysis of the diversity of the predictions for different sampling procedures in comparison to deep ensembles. However, the novelty and the significance of the other contributions are limited (see comments below). Therefore, I consider the paper to be below the acceptance threshold.   

Comments and questions to authors:
1) The practical aspects of different ensembling techniques are not discussed in the paper. While it is known that deep ensembles generally demonstrate stronger performance [1], there is a trade-off between the ensemble performance and training time/memory consumption. The considered alternative ensembling procedures can be favorable in specialized settings (e.g. limited training time and/or memory).

2) It remains unclear to me what new insights does the analysis of the low-loss connectors provide? It is expected (and in fact can be shown analytically) that if the two modes define different functions then intermediate points on a continuous path define functions which are different from those defined by the end-points of the path. This result was also analyzed before from the perspective of the performance of ensembles formed by the intermediate points on the connecting paths (see Fig. 2 right in [2]).
Moreover, I would encourage authors to reformulate the statements on the connectivity in the function space such as: 
-- “We demonstrate that while low-loss connectors between modes exist, they are not connected in the space of predictions.”  (Abstract)
-- “the connectivity in the loss landscape does not imply connectivity in the space of functions” (Discussion)
In my opinion, these claims are somewhat misleading. What does it mean that the modes are disconnected in the function space? Neural networks define continuous functions (w.r.t to both the inputs and the weights), and a connector is continuous path in the weight space which continuously connects the modes in the function space (i.e. a path defines a homotopy between two functions). It is true that two modes correspond to two different functions. However, it is unclear in which sense these functions can be considered to be disconnected. 

[1] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NeurIPS, 2017.

[2] Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of DNNs. In NeurIPS, 2018.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The contribution of the paper is the following two findings: 1. Despite the fact that local minima are connected in the loss landscape the functions corresponding to the points on the curve are significantly distinct. 2. The points along the training trajectory correspond to similar functions. 

Originality and novelty. Both findings do not seem quite new. The first conclusion can be mostly derived from Figure 2 right [1]. Moreover, the difference between functions on the curve in terms of predictions is the main motivation of Fast Geometric Ensembling. The second conclusion is also not quite new and there were several approaches to overcome it e.g. SWA [2]. I appreciate that the authors did a much broader investigation of this phenomena than it was done in previous works. Another drawback is lack of practical implications. It is known that ensembling based on dropout is worse than independent networks, but the main advantage of this and similar approaches is memory efficiency. 

The clarity. The paper is well written, contains all necessary references and is easy to follow. The provided experimental results and supporting plots are also clear and contain the necessary description. The only part that I found a bit confusing is radial plots. I would recommend the authors to add more rigorous description of how they constructed these plots to increase clarity of the paper. Can the authors please also clarify how they derived formulas for the expected fractional difference for f^* and f functions in the section 3.2? 

Overall, it is an interesting paper, but the findings are not quite new.
[1]  Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of DNNs. InNeurIPS, 2018
[2] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Av-eraging weights leads to wider optima and better generalization.arXiv preprint arXiv:1803.05407,2018

review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: This paper is trying to answer the question why ensembles of deep neural networks trained with random initialization work so well in practice in improving accuracy. Their proposed hypothesis is that networks trained from different initializations, although all converge to a low-loss/high accuracy optimum, explore different modes in function space and therefore provide more diversity. To experimentally support their hypothesis, first they show that functions along a single training trajectory are similar, however trajectories starting from different initializations may significantly differ. The difference in function space is based on the fraction of points on which the two functions disagree in terms of their prediction. Second, they use different subspace sampling methods around a single optimum and demonstrate that they are significantly less diverse (low disagreement between predictions) than sampling from independent optima through diversity vs accuracy plots. Moreover, they comment on the recent observation that local optima are connected by low-loss tunnels. They experimentally show that even though low-loss/high accuracy path exists between local optima, these tunnels do not correspond to similar solutions in function space, further supporting the multi-mode hypothesis. The authors compare the relative benefit of subspace sampling, weight averaging and ensembling on accuracy and interpret their findings in terms of the hypothesis. 

Overall, the paper is very well written and provides interesting insights into the multi-modal structure of deep neural network loss landscapes. Even though the hypothesis of the paper is not entirely new and has been touched upon in Fort & Jastrzebski (2019), this paper contributes to the field by providing thorough experimental support and clear exposition of the idea. Therefore, I would accept this paper if the authors provided additional experimental results on a different dataset.

The paper mentions that the trends are consistent across all datasets the authors have explored. However, they only provide results on CIFAR-10 (and a limited set of experiments on ImageNet). Since the contribution of the paper heavily relies on providing experimental verification, it would be important to include at least the diversity vs. accuracy plot for the other datasets they have explored to demonstrate that this phenomenon is not specific to CIFAR-10. 

Additionally, I would like to add a couple of comments on the paper that are not part of my decision, but could potentially improve the paper. 
-The diversity score introduced in the paper is simple and intuitive, however it would be interesting to see whether the results hold across different notions of function space disagreement. 

-It is mentioned in the paper that data augmentation has been used for training the ResNet20 architecture. Would the results change significantly without data augmentation, as it adds another source of randomness to the training procedure.

-Some comments on the figures: in Figure 3 it is very difficult to discern any difference between different shades of red (disagreement values), and in this form the plots are not too informative. Maybe rescaling or a different way of presentation would help. Interpreting Figure 7/a is a bit difficult, probably a 3D plot would be useful to explain the different line sections.

review_title: Official Blind Review #2

Meta-review: Paper https://arxiv.org/abs/1802.10026 (Garipov et. al, NeurIPS 2018) shows that one can find curves between two independently trained solutions along which the loss is relatively constant. The authors of this ICLR submission claim as a key contribution that they show the weights along the path correspond to different models that make different predictions ("Note that prior work on loss landscapes has focused on mode-connectivity and low-loss tunnels, but has not explicitly focused on how diverse the functions from different modes are, beyond an initial exploration in Fort & Jastrzebski (2019)"). Much of the disagreement between two of the reviewers and the authors is whether this point had already been shown in 1802.10026.

It is in fact very clear that 1802.10026 shows that different points on the curve correspond to diverse functions. Figure 2 (right) of this paper shows the test error of an _ensemble_ of predictions made by the network for the parameters at one end of the curve, and the network described by \phi_\theta(t) at some point t along the curve: since the error goes down and changes significantly as t varies, the functions corresponding to different parameter settings along these curves must be diverse. This functional diversity is also made explicit multiple times in 1802.10026, which clearly says that this result shows that the curves contain meaningfully different representations.

In response to R3, the authors incorrectly claim that  "Figure 2 in Garipov et al. only plots loss and accuracy, and does not measure function space similarity, between different initializations, or along the tunnel at all. Just by looking at accuracy and loss values, there is no way to infer how similar the predictions of the two functions are." But Figure 2 (right) is actually showing the test error of an average of predictions of networks with parameters at different points along the curve, how it changes as one moves along the curve, and the improved accuracy of the ensemble over using one of the endpoints. If the functions associated with different parameters along the curve were the same, averaging their predictions would not help performance. 

Moreover, Figure 6 (bottom left, dashed lines) in the appendix of 1802.10026 shows the improvement in performance in ensembling points along the curve over ensembling independently trained networks. Section A6 (Appendix) also describes ensembling along the curve in some detail, with several quantitative results. There is no sense in ensembling models along the curve if they were the same model.

These results unequivocally demonstrate that the points on the curve have functional diversity, and this connection is made explicit multiple times in 1802.10026 with the claim of meaningfully different representations: “This result also demonstrates that these curves do not exist only due to degenerate parametrizations of the network (such as rescaling on either side of a ReLU); instead, points along the curve correspond to meaningfully different representations of the data that can be ensembled for improved performance.”  Additionally, other published work has built on this observation, such as 1907.07504 (UAI 2019), which performs Bayesian model averaging over the mode connecting subspace, relying on diversity of functions in this space; that work also visualizes the different functions arising in this space. 

It is incorrect to attribute these findings to Fort & Jastrzebski (2019) or the current submission.  It is a positive contribution to build on prior work, but what is prior work and what is new should be accurately characterized, and currently is not, even after the discussion phase where multiple reviewers raised the same concern. Reviewers appreciated the broader investigation of diversity and its effect on ensembling, and the more detailed study regarding connecting curves. In addition to the concerns about inaccurate claims regarding prior work and novelty (which included aspects of the mode connectivity work but also other works), several reviewers also felt that the time-accuracy trade-offs of deep ensembles relative to standard approaches were not clearly presented, and comparisons were lacking. It would be simple and informative to do an experiment showing a runtime-accuracy trade-off curve for deep ensembles alongside FGE and various Bayesian deep learning methods and mc-dropout. It's also possible to use for example parallel MCMC chains to explore multiple quite different modes like deep ensembles but for Bayesian deep learning. For the paper to be accepted, it would need significant revisions, correcting the accuracy of claims, and providing such experiments.
Decision: Reject