Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:

- This paper found that combining ensembles and data augmentation can harm model calibration. Inspired by this finding, it proposes a simple correction by only applying mixup to certain classes. Empirical experiments show some improvements. 

Pros:
- The paper is clearly written and easy to follow. The motivation and approach is intuitive.
- Experiments are conducted on both small and large datasets.

Cons:
- The findings do not seem too surprising to me. The vanilla deep neural networks are often in the over-confident regime, so either ensemble or mixup itself seems very effective in improving model calibration. But applying these two together might lead the model into the under-fitting regime. 
- It occurs to me that the proposed AugCAMixup is not good enough. From Table 2 we can tell that it basically sacrifices final accuracy with improvement in ECE. This means that applying mixup to all the classes is more preferrable than applyting to a subset of the classes in terms of accuracy. Why don't the authors consider using rescaling to fix the under confident issue?
- The baselines compared in this submission seem too limited. Some rescaling based methods should be considered, for instance, augmixup + rescaling.
- How's the performance on metrics other than ECE? I think it is important to include other metrics as ECE can hide some problems.
 
------
post-rebuttal update

I appreciate the authors for the responses. While the other reviewers give high reviews about this manuscript, I would keep my original reveiw for the following reasons. 1) This manuscript is incremental in nature. I agree with R3 that "understanding which techniques can be combined and which cannot (and how to fix it) is important for developing the field and feels like a small step to the right direction", but a somewhat unsurprising result lacks technical novelty. In one of the responses the authors wrote "The spirit of this work is to point out that not all data augmentations can be combined well with ensembles." I think this potentially means that the conclusion the authors made on mixup could not even transfer to other data augmentations. This even further limits the contribution of this manuscript. So I guess the above argument from R3 is not convincing, or one could try to study the effect of batch norm and ensemble and wrote another good paper. 2) The empirical performance of the method is limited and thus whether the proposed method is useful is a question.
review_title: limited novelty and performance

final_score: rating: 7: Good paper, accept
review_content: After the discussion, my concerns were fixed. The paper explores the interesting relations of Mix Up and Uncertainty, which is useful and will be the right fit for the conference.

**Summary:**

The work studies how a better calibration of individual members of an ensemble affects the calibration of an ensemble. It is demonstrated that i) better calibration of individual members of the ensemble may lead to the worse calibration of the ensemble predictions ii) this is the case when mix-up / label smoothing are used during training. 

To fix the issue Confidence Adjusted mixup Ensembles (CAMixup) is proposed. The CAMixup is an adaptive mixup data augmentation based on per-class calibration criteria. The core idea of CAMixup is to use powerful (unconfidence encouraging) mixup data-augmentation on examples of overconfident classes, and do not use mixup for the under-confident classes. The confidence criteria are computed once in an epoch.

The empirical results are provided in- and out-of-domain for CIFARs(C) and ImageNet(C).

**The concerns:**

1) ECE is a biased estimate of true calibration with a different bias for each model, so it is not a valid metric to compare even models trained on the same data [Vaicenavicius2019]. In other words, the measured ECE has no guaranty to have something to do with the real calibration but reflects the bias of the measured metric. Other metrics, that are based on histogram estimates have the same problem. Please put extra attention to this concern.

What I suggest is the following:

a. Mesure NLL for in- and out-of-domain data. It seems to be still an adequate (indirect) criterion of calibration, and is an adequate criterion of uncertainty estimation. According to [Ashukha2020], the NLL needs a pre-calibrated model with temperature scaling for in-domain data (called calibrated NLL / calibrated LL). 

b. To use the squared kernel calibration error (SKCE) proposed in [Widmann2019] along with de facto standard, but biased ECE. The SKCE is an unbiased estimate of calibration. There might be some pitfalls of this metric that I'm not aware of, but the paper looks solid and convincing. Also, please put attention to Figure 83 in the arХiv version.

Yes, ECE is the standard in the field, but it is the wrong standard that prevents us from meaningful scientific progress, so we should stop using it.

2) The standard deviation needs to be reported everywhere. Especially the differences between close values like (97.52, 97.52, 97.47 in Table 2) may appear not statistically significant. The same touches Fig 5 and other figures that are reported. Otherwise, it is impossible to stay how solid results are.

**Minor comments:**

1) Maybe it worth to provide plot mean-λi vs epoch to illustrate this "Notice that λi is dynamically updated at the end of each epoch."?

2) Figure 4(a) is done slightly disorderly.

3) In the paper, ECE is measured in percentages. As far as I can tell ECE is dimensionless quantities. It is not clear what is intended.

**Final comment:** I put the "marginally below acceptance threshold" score, but I'm willing to increase it after the update with corrections (and hope that these corrections will be done).  I like the direction and CAMixup, but in-domain results are not very consistent (see Fig 5 (d)), the ECE has uncontrollable model-specific biases that ruin all the presented results. 

[Widmann2019] Widmann D, Lindsten F, Zachariah D. Calibration tests in multi-class classification: A unifying framework. In Advances in Neural Information Processing Systems 2019 (pp. 12257-12267). https://arxiv.org/pdf/1910.11385.pdf

[Ashukha2020] Ashukha A, Lyzhov A, Molchanov D, Vetrov D. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. ICLR, 2020.
review_title: Official Review (Reviewer1)

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This work analyses the interaction between data-augmentation strategies such as MixUp and model ensembles with regards to calibration performance. The authors note how strategies such as mixup and label smoothing, which reduce a single model's over-confidence, lead to degradation in calibration performance when such models are combined as an ensemble. Specifically, all techniques, taken individually, improve calibration by reducing overconfidence. However, in combination they lead to under-confident models and, therefore, worse calibration. Based on this analysis, the author's provide a simple technique which yields SOTA calibration performance on CIFAR-10, CIFAR-10-C, CIFAR-100 and CIFAR-100-C and ImageNet. The authors propose to dynamically enable and disable MixUp based on whether the model is over/under confident on a particular class, as judged on a validation dataset. 

I think this work provides useful insight and a simple and effective solution. Additionally, it is clearly written and very easy and pleasant to read.

The authors may find this concurrent work on ensemble calibration to be relevant: https://openreview.net/forum?id=wTWLfuDkvKp

The only question is have is why the authors think that models are overconfident on hard examples/classes and properly calibrated on easy ones. My intuition would be the opposite - that models are overconfident in areas where there are too few training examples and/or areas where there is no data uncertainty. If there is no data uncertainty then overconfidence would not be a problem. So mainly the issue is due to data sparsity in areas of non-zero data uncertainty. Would be good to expand the discussion. 
review_title: This work provides an interesting analysis of the interaction between data-augmentation strategies such as MixUp and model ensembles with regards to calibration. This work proposes a novel solution to the problem which achieves good results. 

final_score: rating: 7: Good paper, accept
review_content: ##########################################################################
Summary: 

The paper identifies the negative effects on calibration and the robustness of the deep models when data augmentation and the ensembles are combined. The paper proposes a technique that mitigates this drawback and improves the calibration and robustness across benchmarks.

##########################################################################

Reasons for score: 
 
Overall, I vote for accepting. The observations in the paper and the improvements to the state-of-the-art are certainly very encouraging. My minor concerns are on clarity of the writing, references at some places. Hopefully the authors will address these concerns in the rebuttal period. 
##########################################################################


Pros:

+ Overall, the paper is well written, easy to follow and understandable. 

+ The paper is positioned in the body of the literature and does contribute to further improve the state-of-the-art.

+ The pathological effect of augmentations combined with ensembles is an interesting observation. The proposed solution does improve the state-of-the-art both for in-distribution and out-of-distribution data.

+ In section 3.2 the label smoothing experiment is interesting. 

+ The additional experiments in Appendix and the discussion on limitations and potential for future directions is useful and much appreciated.

Cons:

- My first major concern is omitting the key references when the terms are introduced for the first time or some of the assertions. Some of them are listed below. 

- “Ensemble methods are a simple approach to improve a model’s calibration and robustness.” Who found this? This statement requires a citation.

- “For example, the majority of uncertainty models in vision ...” requires a citation.

- “Deep Ensembles” -- need citation, etc. Please fix all such instances

- “and Hendrycks et al. (2020) highlights further improved results in AugMix when combined with Deep Ensembles. However, we find their complementary benefits are not universally true.” From the above sentence, it is not clear what the findings of Hendrycks et al is and for which of their findings the authors in this paper are contradicting with.  I know the paper is about robustness and uncertainty and the authors refer to that. Be clear in the presentation, think that these papers will be read by ML enthusiasts.

- “... poor calibration of combining ensembles and Mixup on CIFAR”, here it is worth to introduce the kind of ensemble(s) that are used in combining.

- DeepEnsembles, BatchEnsemble, MC-Dropout each of them have different ensemble sizes in the experiments. What is the reasoning behind those hyperparameter selections?

- The calibration pathology is observed for ensembles with mixup, is it true for other data augmentations such as rotation, cropping, etc?

- “Ensembles are the among the most known and simple ...” this sentence doesn’t flow well.

review_title: Review: Combining Ensembles and Data Augmentation Can Harm Your Calibration 

Meta-review: This paper analyses the interaction between data-augmentation strategies  and model ensembles with regards to calibration performance. The authors note how strategies such as mixup and label smoothing, which reduce a single model's over-confidence, lead to degradation in calibration performance when such models are combined as an ensemble. They propose a simple solution. The paper merits publication.
Decision: Accept (Poster)