Title: Selective Mixup Helps with Distribution Shifts, But Not (Only) because of Mixup
Venue: NeurIPS 2023
Accepted: no
Keywords: mixup

distribution shifts

OOD generalization

weighted training
Abstract: Mixup is a highly successful technique to improve generalization of neural networks by augmenting the training data with combinations of random pairs. Selective mixup is a family of methods that apply mixup to specific pairs, e.g. only combining examples across classes or domains. These methods have claimed remarkable improvements on benchmarks with distribution shifts, but their mechanisms and limitations remain poorly understood.
 
We examine an overlooked aspect of selective mixup that explains its success in a completely new light. We find that the non-random selection of pairs affects the training distribution and improve generalization by means completely unrelated to the mixing. For example in binary classification, mixup across classes implicitly resamples the data for a uniform class distribution - a classical solution to label shift.
We show empirically that this implicit resampling explains much of the improvements in prior work. Theoretically, these results rely on a "regression toward the mean", an accidental property that we identify in several datasets.

Takeaways:
We have found a new equivalence between two successful methods: selective mixup and resampling. We identify limits of the former, confirm the effectiveness of the latter, and find better combinations of their respective benefits.

Abstract

Context. Mixup is a highly successful technique to improve generalization of neural networks by augmenting the training data with combinations of random pairs. Selective mixup is a family of methods that apply mixup to specific pairs, e.g. only combining examples across classes or domains. These methods have claimed remarkable improvements on benchmarks with distribution shifts, but their mechanisms and limitations remain poorly understood.

Findings. We examine an overlooked aspect of selective mixup that explains its success in a completely new light. We find that the non-random selection of pairs affects the training distribution and improve generalization by means completely unrelated to the mixing. For example in binary classification, mixup across classes implicitly resamples the data for a uniform class distribution - a classical solution to label shift. We show empirically that this implicit resampling explains much of the improvements in prior work. Theoretically, these results rely on a 'regression toward the mean', an accidental property that we identify in several datasets.

Takeaways. We have found a new equivalence between two successful methods: selective mixup and resampling. We identify limits of the former, confirm the effectiveness of the latter, and find better combinations of their respective benefits.

1 Introduction

Mixup and its variants are some of the few methods that improve generalization across tasks and modalities with no domain-specific information [36]. Standard mixup replaces training data with linear combinations of random pairs of examples, proving successful e.g. for image classification [35], semantic segmentation [9], natural language processing [30], and speech processing [21].

This paper focuses on scenarios of distribution shift and on variants of mixup that improve outof-distribution (OOD) generalization. We examine the family of methods that apply mixup on selected pairs of examples, which we refer to as selective mixup [7, 15, 19, 22, 28, 31, 33]. Each of these method uses a predefined criterion. 1 For example, some methods combine examples across classes [33] (Figure 1) or across domains [31, 15, 19]. These simple heuristics have claimed remarkable improvements on benchmarks such as DomainBed [5], WILDS [12], and Wild-Time [32].

Despite impressive empirical performance, the theoretical mechanisms of selective mixup remain 29 obscure. For example, the selection criteria proposed in [33] include the selection of pairs of the 30 same class / different domains, but also the exact opposite. 31

32

33

34

35

36

37

38

39

Original

training

data

{

}

,       ,       ,       ...

{

}

,       ,       ,       ...

Class 1              2                  1                   3

1

⊕

2              1

⊕

3             1

⊕

2              2

⊕

3

More

uniform

Proportions of

classes used

Mixup

across

classes

This raises questions:

What mechanisms are responsible for the improvements of selective mixup?

What makes each selection criterion suitable to any specific dataset?

This paper presents surprising answers by highlighting an overlooked side effect of selective mixup. The non-random selection of pairs implicitly biases the training distribution and improve generalization by means completely unrelated to the mixing . We observe empirically that simply concatenating - rather than mixing - the selected pairs along the mini-batch dimension often produces the same improvements as mixing them. This critical ablation was absent from prior studies.

We also analyze theoretically the resampling induced by different selection criteria. We find that 40 conditioning on a 'different attribute' (e.g. combining examples across classes or domains) brings 41 the training distribution of this attribute closer to a uniform one. Consequently, the imbalances in 42 the data often 'regress toward the mean' with selective mixup. We verify empirically that several 43 datasets do indeed shift toward a uniform class distribution in their test split (see Figure 10). We also 44 find remarkable correlation between improvements in performance and the reduction in divergence 45 of training/test distributions due to selective mixup. This also predicts an unknown failure mode of 46 selective mixup when the above property does not hold. 47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

Our contributions are summarized as follows.

We point out an overlooked resampling effect when applying selective mixup (Section 3).

We show theoretically that certain selection criteria induce a bias in the distribution of features and/or classes equivalent to a 'regression toward the mean' (Theorem 3.1). In binary classification for example, selecting pairs across classes is equivalent to sampling uniformly over classes, the standard approach to address label shift and imbalanced data.

We verify empirically that multiple datasets indeed contain a regression toward a uniform class distribution across training and test splits (Section 4.6). We also find that improvements from selective mixup correlate with reductions in divergence of training/test distributions over labels and/or covariates. This strongly suggests that resampling is the main driver for these improvements.

We compare many selection criteria and resampling baselines on five datasets. In all cases, improvements with selective mixup are partly or fully explained by resampling effects (Section 4).

The implications for future research are summarized as follows.

We connect two areas of the literature by showing that selective mixup is sometimes equivalent to resampling, a classical strategy for distribution shifts [3, 8]. This hints at possible benefits from advanced methods for label shift and domain adaptation on benchmarks with distribution shifts.

The resampling explains why different criteria in selective mixup benefit different datasets: they affect the distribution of features and/or labels and therefore address covariate and/or label shift.

There is a risk of overfitting to the benchmarks: we show that much of the observed improvements rely on the accidental property of a 'regression toward the mean' in the datasets examined.

68

69

70

71

72

73

74

75

76

77

9

78

80

81

82

83

84

85

2 Background: mixup and selective mixup

Notations. We consider a classification model f θ : R d → [0 , 1] C of learned parameters θ . It maps an input vector x ∈ R d to a vector y of scores over C classes. The training data for such a model is typically a set of labeled examples D = { ( x i , y i , d i ) } n i =1 where y i are one-hot vectors encoding ground-truth labels, and d i ∈ N are optional discrete domain indices. Domain labels are sometimes available e.g. in datasets with different image styles [14] or collected over different time periods [12].

Training with ERM. Standard empirical risk minimization (ERM) optimizes the model's parameters for min θ R ( f θ , D ) where the expected training risk, for a chosen loss function L , is defined as:

An empirical estimate is obtained with an arithmetic mean over instances of the dataset D .

Training with mixup. Standard mixup essentially replaces training examples with linear combinations of random pairs in both input and label space. We formalize it by redefining the training risk.

The expectation is approximated by sampling different coefficients and pairs at every training iteration.

Selective mixup. While standard mixup combines random pairs, selective mixup only combines pairs that fulfill a predefined criterion. To select these pairs, the method starts with the original data D , then for every ( x , y , d ) ∈ D it selects a ( ˜ x , ˜ y , ˜ d ) ∈ D such that they fulfill the criterion represented by the predicate Paired ( · , · ) . For example, the criterion same class / different domain a.k.a. 'intra-label LISA' in [33] is implemented as follows:

̸

Paired ( ( x i , y i , d i ) , ( ˜ x i , ˜ y i , ˜ d i ) ) = true iff ( ˜ y = y ) ∧ ( ˜ d = d ) (same class / diff. domain) (4a) Other examples:

̸

3 Selective mixup modifies the training distribution 86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

The new claims of this paper comprise two parts.

Estimating the training risk with selective mixup (Eq. 2) uses a different sampling of examples from D than ERM (Eq. 1). We demonstrate it theoretically in this section.

We hypothesize that the biased sampling of training examples influences the generalization properties of the learned model, regardless of the mixing operation. We verify this empirically in Section 4 using ablations of selective mixup that omit the mixing operation - a critical baseline absent from previous studies.

Training distribution. This distribution refers to the examples sampled from D to estimate the training risk (Eq. 1 or 2) -whether these are then mixed or not. The following discussion focuses on distributions over classes ( y ) but analogous arguments apply to covariates ( x ) and domains ( d ).

With ERM , the training distribution equals the dataset distribution because the expectation in Eq. (1) is over uniform samples of D . We obtain an empirical estimate by averaging all one-hot labels, giving the vector of discrete probabilities p Y ( D ) = ⊕ ( x , y ) ∈D y / |D| where ⊕ is the element-wise sum.

With selective mixup , evaluating the risk (Eq. 2) requires pairs of samples. The first element of a pair is sampled uniformly, yielding the same p Y ( D ) as ERM. The second element is selected as described above, using the first element and one chosen predicate Paired( · , · ) e.g. from (4a-4c). For our analysis, we denote these 'second elements' of the pairs as the virtual data:

We can now analyze the overall training distribution of selective mixup. An empirical estimate is obtained by combining the distributions resulting from the two elements of the pairs, which gives the vector p Y ( D∪ ˜ D ) = ( p Y ( D ) ⊕ p Y ( ˜ D )) / 2 .

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

Regression toward the mean. With the criterion same class , it is obvious that p Y ( ˜ D ) = p Y ( D ) . Therefore these variants of selective mixup are not concerned with resampling effects. 2 In contrast, the criteria different class or different domain do bias the sampling. In the case of binary classification, we have p Y ( ˜ D )=1 -p Y ( D ) and therefore p Y ( D∪ ˜ D ) is uniform. This means that selective mixup with the different class criterion has the side effect of balancing the training distribution of classes, a classical mitigation of class imbalance [10, 13]. For multiple classes, we have a more general result.

̸

Theorem 3.1. Given a dataset D = { ( x i , y i ) } i and paired data ˜ D sampled according to the 'different class' criterion, i.e. ˜ D = { ( ˜ x i , ˜ y i ) ∼ D s.t. ˜ y i = y i } , then the distribution of classes in D∪ ˜ D is more uniform than in D . Formally, the entropy H ( p Y ( D ) ) ≤ H ( p Y ( D∪ ˜ D ) ) .

Proof:

Theorem 3.1 readily extends in two ways. First, the same effect also results from the different domain criterion: if each domain contains a different class distribution, the resampling from this criterion averages them out, yielding a more uniform aggregated training distribution. Second, this averaging applies not only to class labels ( y ) but also covariates ( x ). An analysis using distributions is ill-suited but the mechanism similarly affects the sampling of covariates when training with selective mixup.

When does one benefit from the resampling (regardless of mixup)? The above results mean that selective mixup can implicitly reduce imbalances (a.k.a. biases) in the training data. When these are not spurious and also exist in the test data, the effect on predictive performance could be detrimental.

We expect benefits (which we verify in Section 4) on datasets with distribution shifts, whose training/test splits contain different imbalances by definition. Softening imbalances in the training data is then likely to bring the training and test distributions closer to one another, in particular with extreme shifts such as the complete reversal of a spurious correlation (e.g. waterbirds [24], Section 4.1).

We also expect a benefit on worst-group metrics (e.g. with civilComments [12] in Section 4.4). The challenge in these datasets comes from the imbalance of class/domain combinations in the training data, and previous work has indeed shown that balancing is a competitive baseline [8, 24].

4 Experiments

We performed a large number of experiments to understand the contribution of the different effects of selective mixup and other resampling baselines (see Appendix B for complete results).

Datasets. We focus on five datasets that previously showed improvements with selective mixup. We selected them to cover a range of modalities (vision, NLP, tabular), settings (binary, multiclass), and types of distribution shifts (covariate, label, and subpopulation shifts).

Waterbirds [24] is a popular artificial dataset used to study distribution shifts. The task is to classify images of birds into two types. The image backgrounds are also of two types, and the correlation between birds and backgrounds is reversed across the training and test splits. The type of background in each image serves as its domain label.

CivilComments [12] is a widely-used dataset of online text comments to be classified as toxic or not. Each example is labeled with a topical attribute (e.g. Christian, male, LGBT, etc.) that is spuriously associated with ground truth labels in the training data. These attributes serve as domain labels. The target metric is the worst-group accuracy where the groups correspond to all toxicity/attribute combinations.

Wild-Time Yearbook [32] contains yearbook portraits to be classified as male or female. It is part of the Wild-Time benchmark, which is a collection of real-world datasets captured over time. Each example belongs to a discrete time period that serves as its domain label. Distinct time periods are assigned to the training and OOD test splits (see Figure 10).

Wild-Time arXiv [32] contains titles of arXiv preprints. The task is to predict each paper's primary category among 172 classes. Time periods serve as domain labels.

Wild-Time MIMIC-Readmission [32] contains hospital records (sequences of codes representing diagnoses and treatments) to be classified into two classes. The positive class indicates the readmission of the patient at the hospital within 15 days. Time periods serve as domain labels.

Methods. We train standard architectures suited to each dataset with the methods below (details 157 in Appendix A). We perform early stopping i.e. recording metrics for each run at the epoch of 158 highest ID or worst-group validation performance (for Wild-Time and waterbirds / civilComments 159 datasets respectively). We plot average metrics in bar charts over 9 different seeds with error bars 160 representing ± one standard deviation. ERM and vanilla mixup are the standard baselines. Baseline 161 resampling uses training examples with equal probability from each class, domain, or combinations 162 thereof as in [8, 24]. Selective mixup ( ■ ) includes all possible selection criteria based on classes 163 and domains. We avoid ambiguous terminology from earlier works because of inconsistent usage 164 (e.g. 'intra-label LISA' means 'different domain' in [12] but not in [32]). Selective sampling ( ■ ) is 165 a novel ablation of selective mixup where the selected pairs are not mixed, but concatenated along 166 the mini-batch dimension. Half of the pairs are dropped at random to keep the size of mini-batches 167 constant. Therefore any difference between selective sampling and ERM is attributable only to 168 resampling effects. We also include novel combinations ( ■ ) of sampling and mixup. Code to 169 reproduce our experiments and figures: https://github.com/<anonymized>/<anonymized> . 170

171

172

173

174

4.1 Results on the waterbirds dataset

The target metric for this dataset is the worst-group accuracy, with groups defined as the four class/domain combinations. The two difficulties are (1) a class imbalance (77 / 23%) and (2) a correlation shift (spurious class/domain association reversed at test time). See discussion in Figure 2.

50

60

70

80

waterbirds: OOD worst-group accuracy (%)

Same domain + Diff. class

Diff. domain + Same class

Diff. domain

Same class

Diff. class

Same domain

Diff. domain + Diff. class

Same domain + Diff. class

Diff. domain + Same class

Diff. class

Diff. domain + Diff. class

Diff. domain

Same class

Same domain

Resampling (uniform combinations)

Resampling (uniform classes)

Resampling (uniform domains)

Vanilla mixup

Baseline (ERM)

Figure 2: Main results on waterbirds .

We first observe that vanilla mixup is detrimen- tal compared to ERM. Resampling with uniform class/domain combinations is hugely beneficial, for the reasons explained in Figure 3. The rank- ing of various criteria for selective sampling is similar whether with or without mixup. Most in- terestingly, the best criterion performs similarly, but no better than the best resampling.

■

Baselines

■

Selective sampling without mixup

■

Selective mixup

This suggest that the excellent performance of the best version of selective mixup is here entirely 175 due to resampling . Note that the efficacy of resampling on this dataset is not a new finding [8, 24]. 176 What is new is its equivalence with the best variant of selective mixup. We further explain this claim 177 in Figure 3 by examining the proportions of classes and domains sampled by each training method. 178

Baseline     ---- Simple resampling ----   --- Selective sampling or selective mixup ----

Classes

ERM

1%

4%

22%

73%

Uniform

domains

49%

1%

8%

43%

Uniform

classes

48%

2%

3%

47%

Uniform

combinations

24%

24%

26%

26%

Diff. domain,

any class

49%

1%

8%

43%

Any domain,

diff. class

48%

3%

3%

46%

Diff. domain,

same class

39%

12%

37%

12%

Same domain,

diff. class

36%

38%

13%

13%

4.2 Results on the yearbook dataset

179

The difficulty of this dataset comes from a slight class imbalance and the presence of covariate/label 180 shift (see Figure 10). The test split contains several domains (time periods). The target metric is the 181 worst-domain accuracy. Figure 4 shows that vanilla mixup is slightly detrimental compared to ERM. 182 Resampling for uniform classes gives a clear improvement because of the class imbalance. With 183 selective sampling (no mixup), the only criteria that improve over ERM contain 'different class'. 184 This is expected because this criterion implicitly resamples for a uniform class distribution. 185

64

66

68

70

yearbook: OOD worst-domain accuracy (%)

Resampling (uniform cl.) + Same class

Same domain + Same class

Same class

Diff. domain + Same class

Diff. class

Same domain

Same domain + Diff. class

Diff. domain + Diff. class

Diff. domain

Diff. domain + Diff. class

Diff. class

Same domain + Diff. class

Same domain + Same class

Same domain

Same class

Diff. domain

Diff. domain + Same class

Resampling (uniform classes)

Resampling (uniform domains)

Vanilla mixup

Baseline (ERM)

With selective mixup, the 'different class' criterion is not useful, but 'same class' per- forms significantly better than ERM. Since this criterion alone does not have resampling effects, it indicates a genuine benefit from mixup restricted to pairs of the same class.

■

Baselines

■

Selective sampling without mixup

■

Selective mixup

■

Novel combinations of sampling and mixup

To investigate whether some of the improvements are due to resampling, we measure the divergence 186 between training and test distributions of classes and covariates (details in Appendix A). Figure 5) 187 shows first that there is a clear variation among different criteria ( · blue dots) i.e. some bring the 188 training/test distributions closer to one another. Second, there is a remarkable correlation between the 189 test accuracy and the divergence, on both classes and covariates. 3 This means that resampling effects 190 do occur and also play a part in the best variants of selective mixup. 191

Accuracy (%)

on each OOD test domain

80.5

82.5

65.5

67.5

68.5

70.5

79.0

80.5

87.5

88.5

92.5

94.0

91.5

92.5

86.5

88.0

85.5

87.0

Domain 1

80.5

82.5

Domain 2

65.5

67.5

Domain 3

68.5

70.5

Domain 4

79.0

80.5

Domain 5

87.5

88.5

Domain 6

92.5

94.0

Domain 7

91.5

92.5

Domain 8

86.5

88.0

Domain 9

85.5

87.0

Baseline (ERM)

Variants of selective sampling/selective mixup

Linear fit

Distance between training/test distributions of inputs (top row; average cosine distance) and classes (bottom row; KL divergence)

Finally, the improvements from simple resampling and the best variant of selective mixup suggest 192 a new combination. We train a model with uniform class sampling and selective mixup using the 193 'same class' criterion, and obtain performance superior to all existing results (last row in Figure 5). 194 This confirms the complementarity of the effects of resampling and within-class selective mixup . 195

196

197

198

199

200

201

202

203

204

205

206

4.3 Results on the arXiv dataset

This dataset has difficulties similar to yearbook and also many more classes (172). Simple resampling for uniform classes is very bad (literally off the chart in Figure 6) because it overcorrects the imbalance (the test distribution being closer to the training than to a uniform one). Uniform domains is much better since its effect is similar but milder.

All variants of selective mixup ( ■ ) perform very well, but they improve over ERM even without mixup ( ■ ). And the selection criteria rank similarly with or without mixup, suggesting that parts of the improvements of selective mixup is due to the resampling. Given that vanilla mixup also clearly improves over ERM, the performance of selective mixup is explained by cumulative effects of vanilla mixup and resampling effects . This also suggests new combinations of methods ( ■ ) among which we find one version marginally better than the best variant of selective mixup (last row).

207

41

42

43

arxiv: OOD worst-domain accuracy (%)

Resampling (uniform dom.) + Same class

Resampling (uniform dom.) + Same domain

Resampling (uniform dom.) + Same domain + Diff. class

Resampling (uniform dom.) + Diff. domain + Diff. class

Resampling (uniform dom.) + Diff. domain + Same class

Resampling (uniform dom.) + Diff. class

Resampling (uniform dom.) + Diff. domain

Diff. domain + Diff. class

Same class

Diff. domain

Same domain

Same domain + Diff. class

Diff. class

Diff. domain + Same class

Diff. domain

Diff. domain + Diff. class

Diff. class

Same domain

Same domain + Diff. class

Same class

Diff. domain + Same class

Resampling (uniform domains)

Resampling (uniform classes)

Vanilla mixup

Baseline (ERM)

■

Baselines

■

Selective sampling without mixup

■

Selective mixup

■

Novel combinations of sampling and mixup

To investigate the contribution of resam- pling, we measure the divergence be- tween training/test class distributions and plot them against the test accuracy (Figure 7). We observe a strong correla- tion across methods. Mixup essentially offsets the performance by a constant factor. This suggests again the inde- pendence of the effects of mixup and resampling.

Baseline (ERM)

Selective sampling

Selective sampling (oracle class distribution)

Vanilla mixup

Selective mixup

Resampling (uniform domains)

Resampling (uniform classes)

Linear fit to selective sampling

0

0.5

1

1.5

KL div. between training/test distributions of classes

41

42

43

44

45

46

47

48

49

OOD Test accuracy (%)

The resampling baselines ( ·· ) also roughy agree with a linear fit to the 'selective sampling' points. We therefore hypothesize that all these methods are mostly addressing label shift . We verify this hypothesis with the remarkable fit of an additional point ( ▲ ) of a model trained by resampling according to the test set class distribution, i.e. cheating.

It represents an upper bound that might be achievable in future work with methods for label shift [1, 17].

We replicated these observations on every test domain of this dataset (Figure 14 in the appendix).

4.4 Results on the civilComments dataset

This dataset mimics a subpopulation shift because the worst-group metric requires high accuracy on 208 classes and domains under-represented in the training data. It also contains an implicit correlation 209 shift because any class/domain association (e.g. 'Christian' comments labeled as toxic more often 210 than not) becomes spurious when evaluating individual class/domain combinations. 211

40

50

60

70

civilComments: OOD worst-group accuracy (%)

Resampling (uniform comb.) + Same domain + Diff. class

Resampling (uniform comb.) + Diff. domain

Resampling (uniform comb.) + Diff. domain + Same class

Resampling (uniform comb.) + Diff. domain + Diff. class

Resampling (uniform comb.) + Diff. class

Resampling (uniform comb.) + Same class

Resampling (uniform comb.) + Same domain

Same domain + Diff. class

Same class

Diff. class

Same domain

Diff. domain

Diff. domain + Diff. class

Diff. domain + Same class

Resampling (uniform comb.) + Same domain + Diff. class

Resampling (uniform comb.) + Diff. domain + Same class

Resampling (uniform comb.) + Same class

Resampling (uniform comb.) + Diff. domain + Diff. class

Resampling (uniform comb.) + Diff. class

Same domain + Diff. class

Resampling (uniform comb.) + Diff. domain

Resampling (uniform comb.) + Same domain

Diff. class

Diff. domain + Diff. class

Same class

Same domain

Diff. domain

Diff. domain + Same class

Resampling (uniform combinations)

Resampling (uniform classes)

Resampling (uniform domains)

Vanilla mixup

Baseline (ERM)

Figure 8: Main results on civilComments .

For the above reasons, it makes sense that resampling for uniform classes or combi- nations greatly improves performance, as shown in prior work [8].

With selective mixup ( ■ ), some criterion (same domain/diff. class) performs clearly above all others. But it works even better without mixup ! ( ■ ) Among many other variations, none surpasses the uniform- combinations baseline .

■

Baselines

■

Selective sampling without mixup

■

Selective mixup

■

Novel combinations of sampling and mixup

H

212

213

214

215

216

217

218

219

220

221

222

223

224

4.5 Results on the MIMIC-Readmission dataset

This dataset contains a class imbalance (about 78/22% in training data), label shift (the distribution being more balanced in the test split), and possibly covariate shift. It is unclear whether the task is causal or anticausal (labels causing the features) because the inputs contain both diagnoses and treatments. The target metric is the area under the ROC curve (AUROC) which gives equal importance to both classes. We report the worst-domain AUROC, i.e. the lowest value across test time periods.

Vanilla mixup performs a bit better than ERM. Because of the class imbalance, resampling for uniform classes also improves ERM. As expected, this is perfectly equivalent to the selective sampling criterion 'diffClass' and they perform therefore equally well. Adding mixup is yet a bit better, which suggests again that the performance of selective mixup is merely the result of the independent effects of vanilla mixup and resampling . We further verify this explanation with the novel combination of simple resampling and vanilla mixup, and observe almost no difference whether the mixing operation is performed or not (last two rows in Figure 9).

52

53

54

55

56

57

MIMIC: OOD worst-d. AUROC (%)

Resampling (uniform cl.) + vanilla mixup

Resampling (uniform cl.) + concatenated pairs

Diff. domain + Diff. class

Diff. class

Same domain + Diff. class

Same domain + Diff. class

Diff. class

Diff. domain + Diff. class

Resampling (uniform classes)

Vanilla mixup

Baseline (ERM)

■

Baselines

■

Selective sampling without mixup

■

Selective mixup

■

Novel combinations of sampling and mixup

To further support the claim that these methods mostly address label shift, we report in Table 1 the 225 proportion of the majority class in the training and test data. We observe that the distribution sampled 226 by the best training methods brings it much closer to that of the test data. 227

4.6 Evidence of a 'regression toward the mean' in the data

We hypothesized in Section 3 that the resampling benefits are due to a 'regression toward the mean' across training and test splits. We now check for this property and find indeed a shift toward uniform class distributions in all datasets studied. For the Wild-Time datasets, we plot in Figure 10 the ratio of the minority class (binary tasks: yearbook, MIMIC) and class distribution entropy (multiclass task: arxiv). Finding this property in all three datasets agrees with the proposed explanation and the fact that we selected them because they previously showed improvements with selective mixup in [32].

In the waterbirds and civilComments datasets, the shift toward uniformity ¨ also holds, but artificially. The training data contains imbalanced groups (class/domain combinations) whereas the evaluation with worst-group accuracy implicitly gives uniform importance to all groups.

5 Related work

228

229

230

231

232

233

234

235

236

237

238

Mixup and variants. Mixup was originally introduced in [36] and numerous variants followed [2]. 239 Many propose modality-specific mixing operations: CutMix [34] replaces linear combinations 240 with collages of image patches, Fmix [6] combines image regions based on frequency contents, 241 AlignMixup [29] combines images after spatial alignment. Manifold-mixup [30] replaces the mixing 242 in input space with the mixing of learned representations, making it applicable to text embeddings. 243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

1930

1970

2010

Years (yearbook dataset)

0

0.2

0.4

0.6

Ratio of minority class

(0.5=uniform)

Training domains (ID)

Test domains (OOD)

2008

2014

2019

Years (MIMIC-readmission dataset)

0

0.2

0.4

0.6

Ratio of minority class

(0.5=uniform)

Training domains (ID)

Test domains (OOD)

2007

2014

2019

Years (arxiv dataset)

0.7

0.8

0.9

1

Normalized entropy of class

distribution (1.0=uniform)

Training domains (ID)

Test domains (OOD)

Mixup for OOD generalization. Mixup has been integrated into existing techniques for domain adaptation (DomainMix [31]), domain generalization (FIXED [20]), and with meta learning (RegMixup [23]). This paper focuses on variants we call ' selective mixup ' that use non-uniform sampling of the pairs of mixed examples. LISA [33] proposes two heuristics, same-class/different-domain and vice versa, used in proportions tuned by cross-validation on each dataset. Palakkadavath et al. [22] use same-class pairs and an additional objective to encourage invariance of the representations to the mixing. CIFair [28] uses same-class pairs with a contrastive objective to improve algorithmic fairness. SelecMix [7] proposes a selection heuristic to handle biased training data: same class/different biased attribute, or vice versa. DomainMix [31] uses different-domain pairs for domain adaptation. DRE [15] uses same-class/different-domain pairs and regularize their Grad-CAM explanations to improve OOD generalization. SDMix [19] applies mixup on examples from different domains with other improvements to improve cross-domain generalization for activity recognition.

Explaining the benefits of mixup has invoked regularization [37] and augmentation [11] effects, the introduction of label noise [18], and the learning of rare features [38]. These works focus on the mixing and in-domain generalization, whereas we focus on the selection and OOD generalization.

Training on resampled data. We find that selective mixup is sometimes equivalent to training on resampled or reweighted data. Both are standard tools [10, 13] to handle distribution shifts in a domain adaptation setting, also known as importance-weighted empirical risk minimization (IW-ERM) [25, 4]. For covariate shift, IW-ERM trains a model with a weight or sampling probability on each training point x as its likelihood ratio p target ( x ) /p source ( x ) . Likewise with labels y and p target ( y ) /p source ( y ) for label shift [1, 17], Recently, [8, 24] showed that reweighting and resampling are competitive with the state of the art on multiple OOD and label-shift benchmarks [3].

6 Conclusions and open questions

Conclusions. This paper helps understand selective mixup, which is one of the most successful and general methods for distribution shifts. We showed unambiguously that much of the improvements were actually unrelated to the mixing operation and could be obtained with much simpler, well-known resampling methods. On datasets where mixup does bring benefits, we could then obtain even better results by combining the independent effects of the best mixup and resampling variants.

Limitations. We focused on the simplest version selective mixup as described by Yao et al. [33]. Many papers combine the principle with modifications to the learning objective [7, 15, 19, 22, 28, 31]. Resampling likely plays a role in these methods too but this claim requires further investigation.

We evaluated 'only' five datasets. Since we introduced simple ablations that can single out the effects of resampling, we hope to see future re-evaluations of other datasets.

Because we selected datasets that had previously shown benefits with selective mixup, we could not verify the predicted failure mode when there is no 'regression toward the mean' in the data.

Finally, this work is not about designing new algorithms to surpass the state of the art. Our focus is on improving the scientific understanding of existing mixup strategies and their limitations.

Open questions. Our results leave open the question of the applicability of selective mixup to real situations. The 'regression toward the mean' explanation indicates that much of the observed improvements are accidental since they rely on an artefact of some datasets. In real deployments, distribution shifts cannot be foreseen in nature nor magnitude. This is a reminder of the relevance of Goodhart's law to machine learning [26] and of the risk of overfitting to popular benchmarks [16].

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

References

Kamyar Azizzadenesheli, Anqi Liu, Fanny Yang, and Animashree Anandkumar. Regularized learning for domain adaptation under label shifts. arXiv preprint arXiv:1903.09734 , 2019.

Chengtai Cao, Fan Zhou, Yurou Dai, and Jianping Wang. A survey of mix-based data augmentation: Taxonomy, methods, applications, and explainability. arXiv preprint arXiv:2212.10888 , 2022.

Saurabh Garg, Nick Erickson, James Sharpnack, Alex Smola, Sivaraman Balakrishnan, and Zachary C Lipton. Rlsbench: Domain adaptation under relaxed label shift. arXiv preprint arXiv:2302.03020 , 2023.

Arthur Gretton, Alex Smola, Jiayuan Huang, Marcel Schmittfull, Karsten Borgwardt, and Bernhard Schölkopf. Covariate shift by kernel mean matching. Dataset shift in machine learning , 2009.

Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434 , 2020.

Ethan Harris, Antonia Marcu, Matthew Painter, Mahesan Niranjan, Adam Prügel-Bennett, and Jonathon Hare. Fmix: Enhancing mixed sample data augmentation. arXiv preprint arXiv:2002.12047 , 2020.

Inwoo Hwang, Sangjun Lee, Yunhyeok Kwak, Seong Joon Oh, Damien Teney, Jin-Hwa Kim, and Byoung-Tak Zhang. Selecmix: Debiased learning by contradicting-pair sampling. arXiv preprint arXiv:2211.02291 , 2022.

Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves competitive worst-group-accuracy. In Conference on Causal Learning and Reasoning , 2022.

Md Amirul Islam, Matthew Kowal, Konstantinos G Derpanis, and Neil DB Bruce. Segmix: Cooccurrence driven mixup for semantic segmentation and adversarial robustness. International Journal of Computer Vision , 131(3):701-716, 2023.

Nathalie Japkowicz. The class imbalance problem: Significance and strategies. In Proc. of the Int'l Conf. on artificial intelligence , volume 56, pages 111-117, 2000.

Masanari Kimura. Why mixup improves the model performance. In International Conference on Artificial Neural Networks (ICANN) , 2021.

Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al.

Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning , 2021.

Miroslav Kubat, Stan Matwin, et al. Addressing the curse of imbalanced training sets: one-sided selection. In Icml , volume 97, page 179. Citeseer, 1997.

Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier 322 domain generalization. In IEEE International Conference on Computer Vision , pages 5542323 5550, 2017. 324

Tang Li, Fengchun Qiao, Mengmeng Ma, and Xi Peng. Are data-driven explanations robust 325 against out-of-distribution data? arXiv preprint arXiv:2303.16390 , 2023. 326

Thomas Liao, Rohan Taori, Inioluwa Deborah Raji, and Ludwig Schmidt. Are we learning 327 yet? a meta review of evaluation failures across machine learning. In Thirty-fifth Conference on 328 Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) , 2021. 329

Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift 330 with black box predictors. In International conference on machine learning , 2018. 331

Zixuan Liu, Ziqiao Wang, Hongyu Guo, and Yongyi Mao. Over-training with mixup may hurt 332 generalization. arXiv preprint arXiv:2303.01475 , 2023. 333

334

335

336

Wang Lu, Jindong Wang, Yiqiang Chen, Sinno Jialin Pan, Chunyu Hu, and Xin Qin. Semanticdiscriminative mixup for generalizable sensor-based cross-domain activity recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies , 2022.

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

Wang Lu, Jindong Wang, Han Yu, Lei Huang, Xiang Zhang, Yiqiang Chen, and Xing Xie. Fixed: Frustratingly easy domain generalization with mixup. arXiv preprint arXiv:2211.05228 , 2022.

Linghui Meng, Jin Xu, Xu Tan, Jindong Wang, Tao Qin, and Bo Xu. Mixspeech: Data augmentation for low-resource automatic speech recognition. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 70087012. IEEE, 2021.

Ragja Palakkadavath, Thanh Nguyen-Tang, Sunil Gupta, and Svetha Venkatesh. Improving domain generalization with interpolation robustness. In NeurIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications , 2022.

Francesco Pinto, Harry Yang, Ser-Nam Lim, Philip HS Torr, and Puneet K Dokania. Regmixup: Mixup as a regularizer can surprisingly improve accuracy and out distribution robustness. arXiv preprint arXiv:2206.14502 , 2022.

Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731 , 2019.

Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of statistical planning and inference , 2000.

Damien Teney, Ehsan Abbasnejad, Kushal Kafle, Robik Shrestha, Christopher Kanan, and Anton Van Den Hengel. On the value of out-of-distribution testing: An example of goodhart's law. Advances in Neural Information Processing Systems , 33:407-417, 2020.

Damien Teney, Yong Lin, Seong Joon Oh, and Ehsan Abbasnejad. Id and ood performance are sometimes inversely correlated on real-world datasets. arXiv preprint arXiv:2209.00613 , 2022.

Huan Tian, Bo Liu, Tianqing Zhu, Wanlei Zhou, and S Yu Philip. Cifair: Constructing continuous domains of invariant features for image fair classifications. Knowledge-Based Systems , 2023.

Shashanka Venkataramanan, Ewa Kijak, Laurent Amsaleg, and Yannis Avrithis. Alignmixup: Improving representations by interpolating aligned features. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19174-19183, 2022.

Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David LopezPaz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning , 2019.

Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang. Adversarial domain adaptation with domain mixup. In AAAI Conference on Artificial Intelligence , 2020.

Huaxiu Yao, Caroline Choi, Bochuan Cao, Yoonho Lee, Pang Wei W Koh, and Chelsea Finn. Wild-time: A benchmark of in-the-wild distribution shift over time. Proc. Advances in Neural Inf. Process. Syst. , 2022.

Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Improving out-of-distribution robustness via selective augmentation. In International Conference on Machine Learning , 2022.

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In IEEE/CVF International Conference on Computer Vision , pages 6023-6032, 2019.

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon 381 Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In 382 Proceedings of the IEEE/CVF international conference on computer vision , pages 6023-6032, 383 2019. 384

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond 385 empirical risk minimization. arXiv preprint arXiv:1710.09412 , 2017. 386

Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani, and James Zou. How does 387 mixup help with robustness and generalization? arXiv preprint arXiv:2010.04819 , 2020. 388

Difan Zou, Yuan Cao, Yuanzhi Li, and Quanquan Gu. The benefits of mixup for feature learning. 389 arXiv preprint arXiv:2303.08433 , 2023. 390

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors demonstrate that selective mixup has a similar effect with resampling. This reduces training dataset bias, which may contribute to settings with distribution shifts. Extensive experiments on five datasets were conducted.\n\nSTRENGTHS:\n- A new perspective of (selective) mixup has been discussed.\n- Extensive experiments were conducted on various benchmarks\n- The paper was well written and was easy to follow.\n\nWEAKNESSES:\nThe authors have done a great job in providing an extensive list of experiments. However, I have two major concerns:\n1. The paper's contribution to the body of knowledge is somewhat weak. I believe the gist of the paper is that \"(selective) mixup has a resampling effect\". Although this is a novel and interesting interpretation of the mixup data augmentation method, I feel this is not enough compared to the recent standard level of NeurIPS papers. I believe that a more concrete methodology that reflects the authors' findings is needed to qualify as \"acceptance level\".\n2. The experimental results and their discussions are confusing. While the discussion on the waterbirds dataset says that \"selective mixup performance is entirely due to resampling\", the results on the arxiv dataset says \"selective mixup is affected both by vanilla mixup and resampling\", and results on the civilComments dataset implies that \"resampling is better than mixup\". These different experimental results and interpretations do not conclude the effect of resampling and mixup. Rather, the results seem to imply that \"mixup and resampling are independent methods that may or may not have correlating effects, depending on what dataset is used\". If this is the case, the primary assumption of this paper is undermined.\n\nQUESTIONS:\nIn section 4.1, the authors have mentioned that the similar performance indicates that \"performance of selective mixup is entirely due to resampling\". However, having the same performance level does not mean that one is the cause of the other. Can the authors provide experiments that can demonstrate the causality of the two?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors focus on explaining the working mechanism of selective Mixup in distribution-shift scenarios. They points out an interesting and novel perspective that selective Mixup benefits not (only) from Mixup itself, but rather the resampling effect it induces. They suggest that such a resampling effect can balance the class and/or domain distribution (depends on the pairing criterion) into uniform distribution, which in turn improves the model's performance.\n\nTheoretically, they show that under the different-class criterion of pairing the data, the classes' distribution does become more uniform. Experimentally, they take the worst-group (class or domain or their combination) performance  as the metric, and conduct experiments on several datasets and models. They investigates a variety of algorithms (e.g. ERM, Mixup, conventional resampling, etc.) and pairing criterions (e.g. different class, different domain, etc.). The results confirms their raised point of view on selective Mixup.\n\nSTRENGTHS:\n1. This paper presents an interesting and novel perspective of  undderstanding the mechanism of selective Mixup. That is, instead of focusing on the data interpolation part of Mixup, which is conventionally considered the key of Mixup and any of its variant, this paper indicates that it's the resampling effect, which averages the data distribution, that enables selective Mixup to success.\n\n2. In all the experiments, the paper has fully investigated all the possible sampling criterions. Furthermore, on the basis of these experiments results, this paper combines the algorithms and criterions that give the best performance respectively and obtains a even better result under its own testing metric.\n\n3. All the figures are well displayed and clearly readable.\n\nWEAKNESSES:\n1. This paper is a experiment-driven work. Not many theories are provided to back the main idea up. While Theorem 3.1 shows how certain selective sampling \"uniformizes\" the class distribution, there is no evidence nor explanation of how it affects the covariate distribution. Also, there is no investigation or explanation of what roles (if any) the Mixup operation itself play in selective Mixup.\n\n2. When performing Mixup and selective Mixup training as baselines, the hyperparameter of the Beta distribution $B(\\alpha, \\alpha)$ is not carefully determined, or at least the choosing process is not fully explained. If the Mixup baselines failed to reach their optimal states, then the comparison of them with other methods like resampling and selective sampling w/o Mixup might be unqualified.\n\n3. The testing metric only considers the worst-group performance, but not including the overall performance. Normally practitioners are concerned not only with the short board of the bucket, but also all the boards as an entirety (a trade-off between fairness and overall generalization).\n\nQUESTIONS:\n1. Line 77-78, Equation (3). Here the paper suggests that the interpolation coefficient be drawn from $Beta(2,2)$ distribution. Is that also the setting for the following experiments? Since the conventional choice of the $\\alpha$ is 1, and some works have indicated that large $\\alpha$ may hurt the performance of Mixup. For example, in Figure 2 and line 175-176, the paper implies that selective sampling works similarly whether w/ or w/o Mixup, and that selective Mixup's success is $\\textbf{entirely}$ due to resampling; in Figure 8, it also implies that selective sampling works better than selective Mixup. If the $\\alpha$ in these experiments are not carefully determined, then these comparison results between selective sampling and selective Mixup may be simply due to the sub-optimality of the latter one baseline. I would suggest that whenever we perform Mixup training as baseline, we select the best hyperparameters setting ($Beta(\\alpha,\\alpha)$) such that the performance of Mixup is fully recovered in the first place, otherwise the comparison of Mixup with other methods might be unfair to it.\n\n2. Also the impact of manifold intrusion (if exists in the datasets used in this paper) on the Mixup training would be another interesting factor to consider, since this issue itself is a poison that hurts Mixup's performance. Some other works also have implied that Mixup induces label noises into the data. That is to say, if manifold intrusion does exist in some of the datasets, then what selective sampling actually does is simply erasing these poison by removing Mixup operation itself. In this case, it is unreasonable to suggest that resampling operation (and its effect in averaging data distribution) is the key for the success of selective Mixup. On ther other hand, if manifold intrusion does exist, a large $\\alpha$ may further exacerbate its impact, which could also explain why sometimes w/o Mixup ourtperforms Mixup significantly.\n\n3. The end of line 187. A redundant right bracket.\n\n4. Figure 3, the second plot (uniform domains) and the fifth (diff. domain any class). The percentages on the figures don't add up to 100%.\n\n5. Figure 2. Is there any insight to explain why vanilla Mixup is the $\\textbf{second}$ worst (worse than selective Mixup with diff.domain diff. class criterion)? I was imagining that compared to the latter one, the former one will make the data distribution more imbalanced so it should nevertheless give the worst results.\n\n6. Figure 4. Is there any insight to explain the benefit of same class pairing in Mixup more detailed?\n\n7. CIFAR-10 and CIFAR-100 has some covariate-shift variants, like CIFAR-10-C, CIFAR-100-C, CIFAR-10.1 and CIFAR-10.2. If time and computing resources allow, it would be interesting to see how selective Mixup and selective sampling perform on them. \n\n8. Appendix C, Theorem C.1, Equation (6). The subscript of the summation $\\sum_{i\\neq{j}}^{C}p_j$ could be rewritten as $j\\neq{i}$, which may looks better. Also in both Equations (6) and (7), the typography of the denominator \"$/ (C-1)$\" is a bit off.\n\n9. Line 120-121. Any insight of how certain selective pairing criterions will also have averaging effect on the covariates ($x$)?\n\nLIMITATIONS:\n1. The paper is fundamentally based on experimental investigation, but with little theoretical support. \n\n2. The datasets used doesn't include some of the most popular or commonly used ones like CIFAR or Imagenet. \n\n3. In the experiments, Mixup (and selective Mixup) are taken as baselines, but the value of $\\alpha$ (which supervises the Beta distribution) may not have been carefully determined.\n\n4. I personally imagine that the datasets size and\\or the batch size will also have an impact on the performance of selective Mixup and selective sampling. There might be some room for further investigations in the future.\n\n5. What roles (either positive or even negative) the Mixup operation itself plays in selective Mixup exactly have yet to be fully understood.\n\n6. The metric is limited at the worst-group performance with no information on how the models perform in general on all the testing data.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper performs an analysis of selective mixup techniques, coming to the conclusion that some of the improvements these techniques provide can be obtained by the sampling procedure and not the mixing strategy. Based on some of the insights, variants of selective mixup are sometimes proposed to fit specific settings and shown to work better.\n\nSTRENGTHS:\n- The considered datasets and settings are varied and extensive\n- The paper proposes plausible explanations of the improved performance of selective mixup variants, and performs extensive ablations and creates experimental settings to test these explanations\n- The observations about the roles of re-sampling and regression to the mean are extremely interesting.\n- One of the main interesting points of the paper is carefully studying the forms of shift occurring between the training and the test distributions, and studying how specific forms of selective mixup impact them. This methodology can improve the practitioner's understanding of these techniques and how to improve them (e.g. as shown in some of the cases considered).\n\nWEAKNESSES:\n- As the authors acknowledge, their analysis is limited to the original mixing strategy.   \n- The formatting of the paper is a bit bizarre, with plenty of extremely short paragraphs, misplaced image captions etc. I would recommend the authors to fix this issue, as it makes their work look unprofessional.\n\nQUESTIONS:\n- Although the authors frame this as out-of-scope for their work, at least in the cases in which re-sampling and regression to mean are not sufficient to explain the improved performance, it would be very interesting to see how some of the other mixing strategies (using the citations numbers in the paper, [7, 15, 19, 22, 23, 28, 30, 31, 34]) influence the performance. This would give us a preliminary insight on whether perturbing the mixing procedure (e.g. with CutMix, Manifold Mixup, RegMixup etc.) may significantly affect the performance, and if not, future work should identify other factors.  \n- For Fig. 5 are the authors using Euclidean distances? If so, the result may not be particularly meaningful. Could the authors check the same pattern that occurs with other distance measures in the input space? (e.g. using [1,2])\n\n\n[1] https://github.com/richzhang/PerceptualSimilarity\n[2] https://github.com/Taited/clip-score\n\nLIMITATIONS:\nAdequately Addressed", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper presented an insight that selective mixup (different class or different domain) is actually equivalent to resampling. A simple mathematical proof is presented showing that the mixed up distribution is closer to uniform distribution in terms of label distribution. Extensive experimental results are presented to demonstrate the findings.

STRENGTHS:
* novel understanding on selective mixup
* extensive experiments to demonstrate the understanding

WEAKNESSES:
The finding is nice but seems value is limited, at least for two-class problems, selective mixup obviously leads to uniform distribution. This paper is more about empirical demonstration.

QUESTIONS:
Can I summarize the finding as: selective mixup is mostly equivalent to resampling, and the mixing operation only helps if vanilla mixup helps?

minor question: for waterbird dataset, if class and domain are reversely correlated, how do you sample different classes from the same domain? or is it not perfect correlation?

LIMITATIONS:
not new algorithm, but understanding existing algorithm, the value of the finding seems limited.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07