Tailoring Mixup to Data for Calibration

Authors: Quentin Bouniot, Pavlo Mozharovskyi, Florence d'Alché-Buc
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: mixup, calibration, confidence, robustness

Abstract:
Among all data augmentation techniques proposed so far, linear interpolation of training samples, also called Mixup, has found to be effective for a large panel of applications. 
  Along with improved predictive performance, Mixup is also a good technique for improving calibration.
  However, mixing data carelessly can lead to manifold mismatch, i.e., synthetic data lying outside original  class manifolds, which can deteriorate calibration.
  In this work, we show that the likelihood of assigning a wrong label with mixup increases with the distance between data to mix. 
  To this end, we propose to dynamically change the underlying distributions of interpolation coefficients 
  depending on the similarity between samples to mix, and define a flexible framework to do so without losing in diversity. We provide extensive experiments for classification and regression tasks, showing that our proposed method improves predictive performance 
  and calibration of models, while being much more efficient.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730015950510

thread_type:
official_review

<thread_object>
creator: reviewer_ymx4

<content>
summary:
This paper proposes to improve different mixup methods by adjusting the probability distribution to sample interpolation weights according to the distance between a given sample pair. Using the proposed method, sample pairs that are far from each other are less likely to be mixed. Empirical results on different tasks and data sets demonstrate the effectiveness of the proposed method.

strengths:
The proposed method is clearly introduced and easy to understand

weaknesses:
- The motivation of proposed method can be strengthened
- Some baseline methods seem missing, which can make the empirical comparison not supportive enough

questions:
- Instead of using a similarity kernel introduced in this paper, a simple implementation is to divide sample pairs based on their distances (like the case in Figure 2) and use two different Beta distributions to sample the interpolation weights. How will this baseline perform compared to the proposed method? Such comparison can further verify the motivation and support the proposed method. 
- While the authors have mentioned that non-linear interpolation methods have several disadvantages (larger computational cost, limited application) in section 2.1, some empirical comparison on their performance for image data sets in Table 2-4 should still be necessary. In addition, the authors may also include some of these interpolation methods in Table 6 to verify that their computational cost are muchh larger compared to linear mixup methods. 
- Moreover, can the proposed method be combined with these non-linear interpolation methods as well? I suppose such combination will be straight-forward, as we only need to change the sampling distribution of interpolation weights for different sample pairs. Some discussion (possibly with some empirical results) will be welcome here. 
- I am a bit puzzled by the performance of different methods on ImageNet-A in Table 4. None of them has an accuracy higher than 3%, which seems worse than most methods reported in [1]. Is this due to some discrepancies in experimental setup (e.g., less training epochs)? If so, the authors may need to clarify that such discrepancies are not in favor of their proposed method. 
- Also, how are the baseline methods chosen in Table 6? From my perspective, I suppose we need to compare the time cost of SK Mixup against Mixup or some other baseline methods with rather good performance. Nevertheless, it seems either of the above cases applies, and some explanation may be needed here. 

## References
[1] On Feature Normalization and Data Augmentation. CVPR 2021
</content>
</thread_object>

</thread>
<thread>
creation date:
1730340680755

thread_type:
official_review

<thread_object>
creator: reviewer_fzdf

<content>
summary:
The paper has come up with a concept called manifold mismatch, which is a phenomenon in Mixup that can harm the generalization and calibration performance of the trained models. The paper also show that such a manifold mismatch behavior is correlated to the distance between the data points being paired and mixed in Mixup. Then, by applying a similarity kernel, the paper proposes a variant training algorithm of Mixup that can dynamically adjust the mixing coefficient depending on the distance of the data points being mixed. Finally, the authors have empirically verified the effectiveness of the proposed algorithm on various models and tasks.

strengths:
1. It's a novel idea to summarize several observed phenomenons or properties of Mixup into one unified terminology: manifold mismatch
2. The correlationship between data points pair distance and the occurence of manifold mismatch is carefully derived.
3. The proposed algorithm is meant to balance the diversity and the uncertainty of the virtual examples in Mixup, which is meaningful and helpful

weaknesses:
1. Poor presentation
2. Doesn't clarify the connection between manifold mismatch and calibration. 
3. The core claim that distance impacts manifold mismatch is too simple to be seen, and it's actually under-justified since there seems to be much more necessary conditions of manifold mismatch like data distribution, structure of learned features and manifolds, etc.. In fact, the idea of improving Mixup by forcing more mixtures between closer points is not novel, like k-mixup [1]
4. The idea of dynamically adjust the mixing coeffecient is also not new, like AdaMixup [2].

[1] https://arxiv.org/abs/2106.02933

[2] https://arxiv.org/abs/1809.02499

questions:
1. Line 011 or 012 on page 1. "Along with improved performance", what performance, please make it specified
2. Line 012 or 013 on page 1. "improving calibration and predictive confidence", from my understanding they are the same thing, no need to use "and" here in my opinion.
3. Line 014. Infact in this line the paper has mentioned "calibration of confidence", what is calibration of "confidence", again, in my opinion it's just calibration.
4. Line 019 to 020. Again, please specify what "performance" it is when mentioning "improve performance".
5. Line 054. Here the paper mentions "label noise", can the authors provide a brief description of it like the way they have described manifold intrusion in previous text?
6. Line 140. Again, please specify the term "performance".
7. Is there and related research works about calibration-driven Mixup methods in regression tasks?
8. Line 161. If possible it's much better to turn this title to the next page, it would look better.
9. Line 162 to 169. The notation preliminaries in this paragraph feel a bit too simple and careless. For example, when defining a dataset, one may want to first define a data space and label space, indicating that these spaces are subspaces of some vector spaces of certain dimensions, and then indicate that the training dataset is of some size with the examples being drawn from some data distribution over the data space, etcs.. Putting those details in one single line feels careless. Also, what is $M$? Many would of course just simply take it as the number of classes in the classification problems, but it also needs to be specified in the paper. 
10. Line 162 to 169. There is also some notations that are inconsistent. The paper first indicates that the labels are $M$-dim vectors, then what is the input and output dimension of the encoder $h_\phi$? The way it is presented make the model output $f_\theta(\text{x})$ feels like a scalar, while it should be a vector the same dimension as the labels since $\hat{y}:=f_\theta(\text{x})$.
11. Line 193 to 194. "bounded support $\mathcal{M}_m\subset\mathcal{H}$". Is this also how the manifolds are defined in this paper?
12. Line 194 to 195. To assume that classes are separated, it means all samples belonging to "different classes manifolds" should disjoint, is that correct?
13. Line 208. Why would the mixed points' belonging "to no vlass manifold at all" be a bad thing? In my opinion, when using Mixup for training, especially when the data dimension is high, it's actually mostly the case that the mixed point fall into the "void" areas in the entire data space.
14. Line 212. "the higher the distance between two points, the more likely their convex combination will fall outside of the original manifolds". Here is a counter example. Suppose we have two classes of points in $\mathbb{R}^2$. Suppose their manifolds are two separated line segments on the horizontal axis. If we combine the rightest point from the left manifold and the leftest point from the right manifold, the mixed points will almost all fall into the middle area which is outside both the original manifolds. If we combine the leftest points from both manifold, the fact is, a much bigger proportion of mixed points would fall inside the left manifold. But, the distance between the points in the second case is much larger than that in the first case.
15. Line 215. "Using a Resnet18 trained ...", is it trained using ERM or Mixup?
16. How to choose $\tau_{std}$ and $\tau_{max}$?
17. How does SK Mixup help improve calibration? The derivation of the algorithm only suggests that SK Mixup can help mitigate manifold mismatch, but is manifold mismatch the necessary reason, or even the real reason, of bad calibration?
18. In the process of trading off between diversity and uncertainty, how would calibration and generalization behave? Will they behave like a up-side-down U curve such that there is a sweet spot?
19. The page number of page 7, and the header "under review" line in page 8, there are some strange hyperlinks around the text.
20. Theorem 3.1 (i). "$\lambda\in]\lambda_1,\lambda_2[$", it should be $[\lambda_1,\lambda_2]$ right?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730388684665

thread_type:
official_review

<thread_object>
creator: reviewer_xhfp

<content>
summary:
This paper first demonstrates that distance between the data used in mixup can impact the likelihood of manifold mismatch (a phenomenon where mixed samples lie outside of the class manifolds of the original data used for mixing). It then proposes an efficient framework to mitigate the occurrence of manifold mismatch. The key idea is to dynamically change the distributions of mixing coefficients via a similarity kernel that measures the distance between the mixed samples. Empirical results are provided to demonstrate the effectiveness of the proposed method.

strengths:
- Overall the paper is well written and is easy to follow
- An innovative yet efficient method is proposed to improve mixup so that manifold mismatch likelihood is reduced, and better calibration and accuracies can be achieved
- Adequate empirical results on both classification and regression tasks are provided to demonstrate the proposed method

weaknesses:
- While Theorem 1 shows the existence of manifold mismatch, it does not tell us anything about the assumption (which is used in the paper) that the higher the distance between two points, the more likely their convex combination will fall outside of the original manifolds, and thus, the more likely a model would assign a different label than the original labels of the two points. Some theoretical results in establishing the validity of such assumption would further strengthen the paper. 
- There are some missing papers that could be worth mentioning in the related work. In particular, mixup related works such as [1,2,3]. 
- For the experiments, it is not clear how the proposed method would compare with the method proposed by [1,2,3], and also how would it perform for more complex datasets like ImageNet, ImageNet-C, ImageNet-R and ImageNet-P.

---

[1] Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. AugMix: A simple data processing method to improve robustness and uncertainty. Proceedings of the ICLR, 2020

[2] Soon Hoe Lim, N. Benjamin Erichson, Francisco Utrera, Winnie Xu, and Michael W. Mahoney. Noisy feature mixup. Proceedings of the ICLR, 2022.

[3] Erichson, N. Benjamin, Soon Hoe Lim, Francisco Utrera, Winnie Xu, Ziang Cao, and Michael W. Mahoney. Noisymix: Boosting robustness by combining data augmentations, stability training, and noise injections. Proceedings of the AISTATS 2024.

questions:
- In Eq. (3), there is a -1 inside the exponential. Why not absorb it into $\tau_{max}$? Are there any reasons why it is presented that way? 
- In the classification scenario where the data samples (but not the labels) are noisy and corrupted, thus blurring the true distance between the samples used in mixing, how would the proposed method perform?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730683522861

thread_type:
official_review

<thread_object>
creator: reviewer_zk3c

<content>
summary:
This paper proposes a novel Mixup framework that employs a Similarity Kernel (SK) called SK Mixup to achieve a stronger interpolation between similar points while reducing interpolation otherwise. As a motivation of this study, the authors defined the concept of manifold mismatch, which can negatively impact the calibration of confidence in Mixup. They conducted experimental validation to assess the impact of this phenomenon on the distance between points to mix. Following the presentation of SK Mixup, the effectiveness of the proposed approach in alleviating the manifold mismatch was demonstrated through extensive experiments in classification and regression. Nevertheless, the extension to more complex tasks and theoretical analysis on the regularization effect of SK Mixup still need to be undertaken -- This is why the authors did not receive the highest score in *Contribution*.

strengths:
* The proposed approach was well motivated by targeting the appropriate research problem of calibration-driven Mixup methods in classification and regression: Manifold mismatch. Furthermore, this problem was theoretically defined and proved with the supplement materials.

* The proposed method was in a clear and well-organized manner, and the proof was provided in a manner that supported its conclusions.

* The research problem was validated through experimental results; a comparison was made with the baseline.

* Comprehensive experiments in classification and regression, with a particular focus on performance and computational cost, demonstrated the efficacy of the proposed approach.

weaknesses:
There is no weakness in the paper that would justify its rejection.

questions:
**Question 1.** Why did not SK RegMixup be applied to ViT in Table 4? Is there any potential challenges or limitations of applying SK RegMixup to ViT architectures?

**Question 2.** [line 976-978] In the proof of the first part in Theorem 3.1., whatever the value of $\lambda\_{1}$ is, $\lambda$ can be 0. When $\lambda=0$, then $\tilde{\mathbf{x}}(\lambda)=\mathbf{x}\_{l} \in \mathcal{M}\_{j}$, not $\mathcal{M}\_{i}$, and so does symmetrical case. In my opinion, it would be $\forall \lambda \geq \lambda\_{1},~\tilde{\mathbf{x}}(\lambda)=\lambda \mathbf{x}\_{k}+(1-\lambda)\mathbf{x}\_{l} \in \mathcal{M}\_{j}$. The authors are kindly requested to examine and fix them if there is an error.

---

**Things to improve the paper that did not impact the score:**

* Figure 4 caption: there is no result about *Circles* toy datasets. The mention of it from the caption would be removed if it's not intended to be included

* [line 442] The ECE and AECE of MIT-A in ImageNet-R would be in bold. The authors have to double-check their result highlighting in Table 4 to ensure consistency and accuracy.

*  In Table 4, is there any reason why ECE and AECE are exactly same in OOD settings? The authors are kindly requested to explain why ECE and AECE are identical in OOD settings or, if this is unexpected, verify whether there might be an error in the reporting or calculation of these metrics.

* The experimental results show that the SK RegMixup is an effective method. However, Table 6 does not provide the efficiency of the SK RegMixup in terms of computational cost. It is recommended to include computational cost metrics for SK RegMixup in Table 6, similar to what they've provided for other methods.

* [line 879] Typo: cccross
</content>
</thread_object>

</thread>
<thread>
creation date:
1732290753702

thread_type:
official_comment

<thread_object>
creator: reviewer_fzdf

<content>
comment:
My apologies, I mistook the ending date of the discussion period. I have undone the changes I had made.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732541519898

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revision of the paper and summary of changes

comment:
We thank all the reviewers for their reviews and their insightful comments about our work. We uploaded a revision of the manuscript to include your suggestions. All modifications in the text are colored in blue. Here is a summary of the changes:
- We improved the presentation and clarified some sentences.
- We combined old Figure 1 and Figure 2 into a single Figure using subfigures. 
- We added a novel theoretical result (Theorem 3.2) supporting our experiments in Figure 1a, that we derived as a corollary of Theorem 5.1 from Liu et al. [A]. This result shows that the likelihood of assigning a wrong label with Mixup is lower-bounded by the distance (total variation) between the original manifolds. The proof is in Appendix F.2.
- We added missing baselines for ViT architectures on Imagenet in Table 4.
- We added in Appendix L a comparison with CutMix [B] and with discussions around combining our method with CutMix. 
- We added time comparison with all baselines in Table 6 for Image classification. The full comparison on regression tasks and with non-linear mixup (CutMix) was moved to Appendix M

[A] Liu et al. "Over-training with mixup may hurt generalization." ICLR, 2023  

[B] Yun et al. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV 2019.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a new Mixup method that is able of improve both accuracy and calibration. Specifically, the authors propose a Similarity Kernel to parameterize the distribution of interpolation coefficients, making Mixup more data-centric. The idea is both novel and significant. Extensive experiments demonstrate that this approach improves accuracy and calibration across a variety of tasks, including classification and regression.

The reviewers are positive, noting that the method is well motivated, theoretically justified, and supported by strong experimental validation. Major concerns were addressed during the rebuttal phase, and the paper is clear and enjoyable to read. One possible extension is to consider a non-linear or latent space similarity kernel, for instance, employing a pre-trained autoencoder to compute similarity in latent space.

I recommend to accept this submission.