Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents a combinatorial approach to controlling and improving the effects of diversity; a relevant and interesting feature in Bayesian networks. The approach is general in the sense that it involves minimization of an f-divergence, for any f. The approach is mainly compared to standard deep ensembles, i.e. without controlling diversity. Additionally, the paper provides a theoretical result with approximation guarantees, which might be useful outside this specific application.
main_review: PROS

1. The approach targets diversity, a relevant and interesting phenomena, and gives a technically non-trivial algorithm for controlling it. Especially, I like the connection to the empirical risk in Eq. (1) and the flexibility of the method (it works for any f-divergence). To me it seems the authors expertize in combinatorial optimization.

2. The results seem convincing that the proposed method can outperform standard deep ensembles in OOD tasks. Moreover, Figure 3 is a nice representation of the OOD detection results.

3. The Related Work section and the literature study in the introduction are dense. The authors have clearly spent time probing the field. These sections are nice to read and are useful for navigating in this evolving domain (although I miss one reference, see 2. below).


CONS

1. My main concern is regarding accessibility and clarity of how the method is described. As this is posed as a competing method/framework to standard deep ensembles, the paper should be accessible to practitioners and researchers with a broad technical background. As such, the concepts submodularity, supermodularity and modularity, as well as their definitions in the paper, need to be contextualized.

For instance, I have been working a lot with deep ensembles and Bayesian deep learning, and am interested in the method proposed in the paper, but I have a hard time understanding the motivation of using the modularities? I see that they help devise the method theoretically, but what are f, V, B and A in Definition 1? As an example of contextualization, simply state what f is in this setting. All I understand about f right now is that it is a mapping from a binary input of dimension V to the real line. Also, is f in Def. 1 the same f as in f-divergence? If not, please consider distinguishing them.

In my opinion, the delivery of the method in, especially, sections 2.2 and 3.1 is not easy to follow. These two sections essentially contain enumerations of definitions and I have a hard time finding motivations for them. As far as I am aware, you do not reference the definitions or the theorem in the text.

2. Although the sections Related Work and Introduction are thorough, I think [1] should be included. In [1] they correlate the ensemble components by letting them share weights. Your objective function also correlates the ensemble components. In fact, I think the results in the paper would benefit a lot from comparing with [1] *and*, e.g., [2]. [2] since, as is pointed out in Sec 5, it is "closely connected". Not comparing with POVI methods overall is not well motivated, in my opinion.

3. How did you obtain Figure 1? It is not obvious to me that (a) and (b) would occur. It seems speculative and unmotivated as is (this alone makes my "Correctness" score = 3).

4. The paper has an OK structure in the majority of the sections (especially Sec. 1, 5, 6 and 7) and the language is generally good. However, there are many disturbing typos which could easily have been caught by reading through the paper.

[1] "Training independent subnetworks for robust prediction", Havasi et al., ICLR 2021
[2] "Repulsive deep ensembles are bayesian.", Francesco D’Angelo and Vincent Fortuin. NeurIPS 2021
summary_of_the_review: I am certain that the proposed method can indeed be useful and of interest for many in the field, and the method appears to be largely novel. However, I am concerned that it will not be very impactful, the way it is delivered now. To be clear, I don't believe that the mathematical approach is necessarily complicated. However, the provided explanations are over-complicated, and the approach is insufficiently motivated/contextualized.

To address this concern, I suggest that, for instance, redundant definitions (as Def. 4?) are moved to the Appendix (if at all needed). Instead, there should be some paragraphs in Section 2 that puts concepts like submodularity in context. Can you map it to a concept in Bayesian DL?

Regarding the baselines, I think it is important to compare the method to more recent advances in order to fully grasp its capabilities. As is shown in the Related Work, the field has evolved a lot since deep ensembles were first proposed. Question to the authors: how come more baselines were not considered?
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This proposes studies ensemble construction from a combinatorial point of view. Analysis are conducted in function space, where the intuition are from and algorithm are developed. Specifically, it starts by formally defining the sub-modularity then proving f-divergence for a distribution and a mixture of kernels to be super-modular function. A greedy maximization of f-divergence is given and its implementation for approximating Bayesian posterior are presented. The experiments on synthetic data show the ability of covering the posterior of the proposed method, compared with the deep ensemble approach. It also conducts experiments on real-world data for OOD detection tasks. 
main_review: In general, the paper provides a novel perspective for constructing the ensemble, connecting f-divergence between a distribution and a kernel mixture with sub-modular functions. Submodularity is useful in exploration, e.g., submodular set function maximization. The general idea is novel and intuitive, which could be helpful for the research in deep ensemble.

The execution of the implementation is solid in general. Some findings and analysis are interesting, e.g., a better approximation factor compared with theoretical guaranteed factor could be achieved for f-divergence. I have the following questions or comments before making the final decision:
1. Could you provide a high level explanation of why sub-modularity might be helpful for constructing the ensemble before introducing the definitions, theorems and algorithms? Jumping to this functional view after brief review of previous works might be too abrupt.
2. How are the mode-seeking and mean-seeking ability of the proposed method justified theoretically or empirically?
3. There are a few approximations like using the point estimate for the reverse KL. How does it affect the general performance and how is this quantified?
4. It is claimed the heuristic techniques like batchensemble or hyper parameter ensemble could be used adjunct with the proposed method. However, those methods are scalable to large datasets like ImageNet and could be computed efficiently. I wonder if the proposed method could be extended to ImageNet level datasets.
5. How is the performance of robustness under adversarial attack?
summary_of_the_review: This paper is with interesting idea and solid execution. I recommend a weak accept before the questions are answered.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel and principled method to make Deep Ensemble Bayesian. It minimizes an f-divergence between the true posterior and a kernel density estimator fitting the functions associated with the ensemble candidates. The authors formulate the learning as a submodular problem and propose to use a greedy approach to solve it. The resultant learning objective consists of a novel diversity term. Some out-of-distribution detection results in computer vision demonstrate the efficacy of the proposed method.
main_review: # Strengths
- The problem of making Deep Ensemble Bayesian is important for the community, and the proposed method seems to be a viable approach to solve this problem.

- To my knowledge, the idea of rephrasing the Bayesian inference problem as a submodular one is conceptually novel.

- The theoretical justification is impressive and interesting.

# Weaknesses & Concerns

- It seems that, eventually, you use the reverse KL divergence as an instantiation of the f-divergence, and develop the practical training loss Eq 12 and Eq 14. 
Though your modeling is applicable to arbitrary f-divergence, you have not demonstrated the implications of such universality.

- Eq 14 is closely related to the function-space POVI (Wang et al., 2019). The difference is that you train these ensemble candidates sequentially with weight decay while function-space POVI trains them in parallel without weight decay? I would like to see an empirical/theoretical clarification of the connections/difference between these two approaches.

- Although I know the main contribution of this work is on a new theoretical framework for Bayesian Deep Ensemble, I still want to see that this works can beat Deep Ensemble significantly in more scenarios. The current performance gains in Table 1 are marginal.
summary_of_the_review: This paper has done a great job in making Deep Ensemble Bayesian theoretically, despite being short of empirical verification. So, currently,  I recommend a weak acceptance for it.
review_title: null

Meta-review: The authors present a new framework to make deep ensembles provide better coverage of the posterior and be less reliant on initialisation. The authors generally did a good job presenting their approach, avoiding dubious claims that deep-ensembles are non-Bayesian, and instead focusing on ways in which deep ensembles can be improved, practically and theoretically. It is worth noting in a revised version, however, that many approximate inference procedures do not have theoretical guarantees. The claim that deep ensembles have "arbitrary bad approximation guarantees" is vague and appears to single them out in a way that could confuse the reader. Regarding priors, it is also worth noting that Wilson & Izmailov (2020) provide evidence that the prior in weight space induces a prior in function space with useful properties, although the prior can be improved.

The authors do a good job of responding to reviewers, and describing limitations. Ultimately, however, the general opinion was not swayed to accept. In addition to reviewer concerns, the experimental evaluation could be substantially improved. There are several procedures that build on deep ensembles to capture uncertainty within modes. How does this procedure compare? Why are no likelihood evaluations considered? What about accuracy? In its present form, it's unclear what practical value the contributions are providing, besides possibly better OOD detection, but even that direction is explored in a relatively limited way. It could also be interesting to measure the distance of the predictive distribution to a good proxy for the Bayesian model average. Overall, there are the raw ingredients of a good paper here, and the authors are encouraged to continue with this work.
Decision: Reject