Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper quantifies a notion of diversity for deep ensembles that facilitates efficient estimation. The authors show that it is sufficient to enforce conditional independence on the output distributions of the classifiers. This leads to their main contribution concerning the regularizing metric: conditional mutual information (CMI), efficiently computed in classification problems. The authors name this approach Conditionally Independent Deep Ensembles (CoDE).  The authors evaluate CoDE on benchmark datasets for shortcut learning.
strength_and_weaknesses: Strength:
- The main strength of this paper lies in the clear exposition and development of CoDE.
- The definitions of invariance and diversity specific to CoDE are novel.
- The CoDE optimization problem is new, and its solution is efficient.  
- The numerical results are promising. 

Weakness:
- Novelty of the CoDE objective is unclear in light of the earlier work on conditional mutual information.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and novel.
summary_of_the_review: The authors' main contribution is the development of CoDE approach and the objective for efficient learning in CoDE.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper discusses a method for learning diverse ensembles “Conditionally Independent Deep Ensembles (CoDE)” and benchmarking its performance with shortcut learning datasets (i.e. ColoredMNIST, Waterbirds). The main objective is to enforce variability of the signals picked  avoiding to rely always on the same and/or strongest signal. 

Authors aim to enforce conditional independence  on the output distributions (there are good properties of doing so as stated by authors i.e. being computationally cheaper). Also authors focus on enforcing confident predictors. That takes the form of two loss components: 

(i) CMI - a conditional mutual information component that is computed on pairwise predictors/networks
(ii) R - a “confident”-prediction regularization

Both are controlled by scalar factors to weight their importance.

In terms of literature counterparts, authors reference two main papers: Pace et al. (2020) and Rame & Cord (2021) 
strength_and_weaknesses: Results on the ColoredMNIST dataset (table 1) raises some questions to me (i) does the train column refer to metrics of testing on the train set? and if so, does it show a tendency of ‘competing’ methods to overfit? -(ii) The difference of performance with TC-Ensemble Pace et al. is small, is there perhaps a computational benefit in comparison? (iii) the work by Pace et al. (2020) is included in the table but Rame & Cord (2021) not – being one of the closest references it would help to see its benchmark as well, or understand why it may not be applicable.

The model has a few possible variations: being sequential/joint, which is addressed by results in table 2. However, the two proposed loss components are not further studied beyond the hyperparameter comments on the appendices.

In terms of the data generation process or the impact of enforcing variability of the ensemble learnt it is not further linked to the results i.e. (i) is it a valid assumption or (ii) does the classifiers learnt show signs of being independent.

clarity,_quality,_novelty_and_reproducibility: The paper is well written. The data generation process description is detailed and informative. The notation is consistent.


summary_of_the_review: Overall, I initially recommend rejecting this submission as results are not very strong in showing performance or computational cost improvement.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to learn a set of classifiers to take care of different predictive signals in the dataset. They propose that the "diversity" of the ensemble is important and the conditional independence is an effective way to realize this goal.
strength_and_weaknesses: Pros:
* The proposed fast adaption method for ensemble learning can addresses distribution shift and shortcut learning simultaneously. These two issues are common and realistic in practice.

Cons:
* Based on the description in this paper, I find the separate predictive signals are very similar to multi-view learning. Both of them are to use the different perspectives of a dataset to perform learning tasks. More discussions about the similarities and differences between these two settings are required. If the methods of multi-view learning could be extended to this problem.

* The motivations are not clear. For example, they claim that considering the independence of latent factors conditioned on labels is better. However, I cannot get its advantages over unconditioned independence from this paper. "Dimensions of z can be dependent by virtue of their correlation to y" may be a reason. However, this explanation is too abstract, and more high-level analyses and practical examples will be better. As a way to realize the conditional independence, why do they choose the conditional mutual information? What are its advantages? If other metrics that are related to independence would make the method fail?

* The experiment design is problematic. The title of this paper emphasize the "distribution shift". However, in experiments, they mainly verify the effectiveness for shortcut learning,  and the distribution shift is not verified specifically. They should perform additional experiments on the datasets of domain adaptation or OOD generalization. Besides, due to the similarity with multi-view learning, comparing CoDE with the corresponding methods is necessary.

clarity,_quality,_novelty_and_reproducibility: The clarity needs to be further improved. The quality and originality are fine.
summary_of_the_review: Please refer to Strength And Weaknesses.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a method to (1) learn a diverse deep ensemble where the diversity among the members is ensured by regularizing conditional mutual information on the output distributions of each pair of members, (2) do a fast adaptation of the best model from this ensemble, when given a small amount of training data from a different distribution (in particular, the paper considers the shortcut learning, or spurious correlation problem, as an example of distribution shift).

Although the paper's basic idea is interesting, the reviewers pointed out several issues with the paper. In particular, (1) although it's not the most critical issue, from the title the paper appears to be about distribution shift and OOD generalization; however, it only considers a very specific type of distribution shift (shortcut learning/spurious correlation), (2) experimental results being weak in terms of performance as well as computational gains, (3) conditional mutual information based regularizers used in several prior works, which undermines the novelty further (even though the paper applies it on the output distribution as opposed to on latent representations).

The authors provided a response to the issues raised by the reviewerers. However, the concerns still linger due to several reasons, some of which are listed below:

- The original submission gave a somewhat misleading impression as proposing a diverse ensemble based method to handle the general distribution shift, whereas it only tackles the shortcut learning problem. The authors clarify this in the rebuttal; however, it raises the question about the positioning of this work w.r.t. the existing work on learning deep (or diverse deep) ensembles that tackle general distribution shift as well as shortcut learning/spurious correlations.

- The paper doesn't compare with a very relevant baseline - Rame & Cord (ICLR 2021) - "DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation". The authors mention the lack of an available implementation as the reason for not including this baseline and say that Pace et al (2020; unpublished) is an equivalent baseline. However, this is not convincing. Rame & Cord (2021), provided a more detailed and rigorous study, on various metrics, on several datasets. Even if the implementation is not available, an attempt could be made to compare with the results reported in their paper, following the same experimental settings.
   
- For deep ensembles, the idea of diversity regularization via the output distribution is not new. For example, please see "Diversity regularization in deep ensembles" (Shui et al, 2018). Even though they don't use conditional mutual information, they do use the outputs of pairs of ensemble members to enforce diversity. Also see "Improving Adversarial Robustness via Promoting Ensemble Diversity" (Pang et al, 2019) and "Improving robustness and calibration in ensembles with diversity regularization" (Mehrtens et al, 2022) which discuss several regularizers on the output distributions.

- Given that other prior works on diverse deep ensembles are not limited to shortcut learning and explore their usefulness for general OOD generalization, the paper should also explore the same and compare with such methods on these problems. Without such comparisons, the paper's contribution/impact will be very limited (especially because the basic idea of using conditional mutual information is not very novel in itself).

Considering the above points and the comments from the reviewers, the paper in its current form does not appear ready for acceptance. The authors are advised to consider these points to improve the work and resubmit to another venue.
justification_for_why_not_higher_score: Please refer to the reviews as well as the meta-review. The paper lacks on various accounts - novelty, experiments, missing related work, as well as unclear positioning.
Decision: Reject