Enhancing Minority Classes by Mixing: An Adaptative Optimal Transport Approach for Long-tailed Classification
Authors: Jintong Gao
He Zhao
Zhuo Li
Dan dan Guo
Venue: NeurIPS 2023 poster (2023)
Keywords: Long-tailed Classification
Optimal Transport
Image-mixing
Semantic Similarity

Abstract: Real-world data usually confronts severe class-imbalance problems, where several majority classes have a significantly larger presence in the training set than minority classes. One effective solution is using mixup-based methods to generate synthetic samples to enhance the presence of minority classes. Previous approaches mix the background images from the majority classes and foreground images from the
minority classes in a random manner, which ignores the sample-level semantic similarity, possibly resulting in less reasonable or less useful images. In this work, we propose an adaptive image-mixing method based on optimal transport (OT) to incorporate both class-level and sample-level information, which is able to generate semantically reasonable and meaningful mixed images for minority classes. Due to
its flexibility, our method can be combined with existing long-tailed classification methods to enhance their performance and it can also serve as a general data augmentation method for balanced datasets. Extensive experiments indicate that our method achieves effective performance for long-tailed classification tasks. The code is available at https://github.com/JintongGao/Enhancing-Minority-Classes-by-Mixing.

Enhancing Minority Classes by Mixing: An Adaptative Optimal Transport Approach for Long-tailed Classification
Jintong Gao', He Zhao?, Zhuo Li Dandan Guo School of Artificial Intelligence, Jilin University CSIRR's Data61
Shenzhen Research Institute of Big Data "The Chinese University of Hong Kong, Shenzhen he zhao@ieee.. org guodandan edu.
gaojt200mails. jlu. edu, on 221019088@1 ink. cuhk. edu. en
Abstract
Real-world data usually confronts severe dlass-imbllane problems, where several majority classes have significantly larger presence the training than minority classes. One effective solution is using mixup-based methods to generate synthetic samples to enhance the presence of minority classes. Previous approaches mix the background images from the majority classes and foreground images from the minority classes random manner, which ignores the sample-level semantic similarity, possibly resulting in less reasonable or less useful images. In this work, we propose an adaptive image-mixing method based on optimal transport (OT) to incorporate both class-level and sample-level information, which is able to generate semantically reasonable and meaningful mixed images for minority classes. Due to its flexibility, our method can be combined with existing long-tailed classification methods to enhance their performance and it can also serve as general data augmentation method for balanced datasets. Extensive experiments indicate that our method achieves effective performance for long-tailed classification tasks. The -iinoiMty-Claassetee
code is available at by-Mixing
1 Introduction
Large-scale balanced datasets play a vital role in the remarkable success of deep neural networks for various tasks. However, datasets in many real-world applications often exhibit unexpected long- long-tailed distributions where most of the data belongs to several majority classes while the rest spreads across lots of minority classes. The model trained such long-tailed dataset will be biased toward majority classes, leading to poor generalizations about minority classes.
Re-weighting [1 2 5 6 7), over-sanpling 88 10 12). under sampling [3 14 15 16),
data augmentation 17 18 19
20 21 22 23), two-stage methods [24 25 26 27]. and other methods 22
28 29 are common solutions to the long-tailed problem. Among them, over-sampling and data augmentation aim to balance the data distribution by oversampling or generating closely related minority classes. As the representative data augmentation techniques, mixup and its variants [30] (2) have performed satisfactorily in the computer vision fields, whose key idea is constructing mixed samples by performing linear interpolations between data/features and corresponding labels. Considering mixup-based methods are designed for balanced data, applying them to long-tailed classification without any adjustments may ignore the specificity of long-tailed data distribution. Recently, some novel mixup- based methods for the long-tailed problem have been explored (2 33
*Coressonding author. This work was supported by NSFC (62306125).
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Foreground
Mixed Samples
Foreground
Mixed Sumples
Panda
Scashore
Hutterily
CMO
Frelis
Taxicab
Grey Whale
Drancttary
Eagle
Panda
Digong
Orea
Ours
Fub Arctos
Black Bear
Grey Whale
Sea Lion
Figure 1: Visualization of mixed samples created by CMO and ours(OTmix), respectively.
34 21J. As a representative example, Context-rich Minority Oversampling (CMO) 20) generates minority-centrie images by mixing the background and foreground images, where background images are mainly from the majority classes while foregrounds are mainly from minority classes. The critical concept of CMO is to paste image from minority class (foreground) onto rich-context images from a majority class (background). Despite the effectiveness of CMO, it cannot be guaranteed that all the generated images are realistic and reasonable by randomly mixing the background and foreground images. For example, we visualize mixed samples created by ours and CMO in Fig. [ where we take "Panda" and "Grey Whale" two foreground images. CMO generates unreasonable mixed samples since pandas are hardly seen in the sky, the beaches, or the streets, and grey whales rarely appear in the desert, the flowers, the sky. Therefore, instead of arbitrarily pairing the foreground and background images, it is essential to consider the semantic distances between the foreground and background images generate more meaningful and advantageous samples.
In this work, we propose a more adaptive image-mixing method based on optimal transport (OT), which is a powerful tool for computing the distance between two distributions under the form of multiple sampling (35]. We consider an empirical distribution Paack over the background images and an empirical distribution Qiore over the foreground images, where the former more likely from the majority and the latter from minority classes. To learn the semantic similarity between background and foreground images, we formulate the learning of the similarity the OT problem between Prack and Qtore. In this way, we can view the learned transport plan as the similarity between foreground and background samples, which can be naturally used guidance to select the most relevant background image for each foreground image. Due to the importance of cost function in learning the transport plan for OT, we further design the cost function using the sample-level representation and class-level label information. Hence, the rich context of the majority classes as background images can be better transferred the minority classes foreground images for providing adaptive way to learn the similarity between them and generate more semantically meaningful mixed images. Interestingly, our proposed OTmix can not only be combined with existing long-tailed methods but also be used as a new image- image-mixing strategy for balanced datasets.
Our main contributions summarized follows: (1) We propose a novel image-mixing method for generating reasonably mixed samples for long-tailed classification, where we introduce a distribution over the background images mainly from majority classes and another distribution over the foreground images mainly from minority classes, (2) By minimizing the OT distance between these two distributions, we view the learned transport plan as similar in guiding the image-mixing process. (3) We design the cost function based on the feature information and confusion matrix to learn the desired transport plan. Extensive experiments demonstrate the effectiveness of our method for long-taile tailed classification and also balanced classification.
Related Work
Data Processing Learning relatively balanced classes from the data perspective is an effective solution to the long-tailed problem [36). which can be roughly divided into over-sampling, under- sampling, and data augmentation. Since our proposed method is related to the over- wver-samping and data augmentation methods, we describe them in detail below. Over-sampling aims to emphasize the
minority classes and increase the instance number of the minority classes [20 g 9.. For example, the classical Synthetic Minority Over-iagp Technique (SMOTE) [12] uses the interpolation between a given minority sample and its nearest minority neighbors to create new samples. Data augmentation another way of data processing to compensate the minority classes by generating and synthesizing new samples [18 23 For example, Major-to-minor Translation (M2m) [17) defines an optimization phase to augment minority classes via translating samples from majority classes. The Meta Semantic Augmentation (MetaSAug) approach (24) is proposed to perform effective

---

semantic data augmentation by learning more meaningful class-wise covariance. Sample-Adaptin Feature Augmentation (SAFA) [19] aims extract diverse and transferable semantic directions from majority classes, and adaptively translate minority-class features along extracted semantic directions
for augmentation.
In addition, mixup- baaxuppaseeddd methods 32 37 38 30 39
31 40
42). the effective data augmentation
methods in balanced datasets, have been developed to solve the long-tailed problem recently and are closely related to our work. For example, Mixup Shifted Label-aware Smoothing Model (MiSLAS) [77 uses mixup in its Stage-1 training without any adjustments. Rebalanced Mixup (Remix) 22T provides a disproportionately higher weight to the minority class when mixing two samples. Uniform Mixup (UniMix) (22) adopts a minority-favore mixing factor to encourage more majority-minoriyy pairs occurrence and is further combined with the Bayes Bias (Bayias) caused by the inconsistency of prior. Targeted copy-paste augmentation (43) aligned with the novel domain enhances out-of- domain robustness on long-tailed camera trap dataset. (44) improves minority class performance from synthetic data by contrasting night and day backgrounds. To mitigate the overfitting issue in subpopulation shift, Uncertainty-awar Mixup (UMIX) (33) adopts the sampling uncertainty to reweight the mixed samples. CMO 20 constructs mixed samples between the background images more likely from the majority classes and foreground images more likely from the minority classes. However, it ignores the semantic similarity between images by randomly mixing them. Different from CMO, formulate the mmage-mixing problem the OT problem between two distributions and use the learned transport plan as guidance to mix images, providing a general and adaptive image-mixing method for long-tailes learning.
Loss Function Engineering Designing effective training objectives is another solution to fight Margin Loss (LDAM) T is proposed from the view of the generalization error bound, and LDAM-DRW adopts a deferred class-level re-weighting method. Balanced Softmax (BALMS) (2) accommodates the label distribution shift between training and testing and proposes a Meta Sampler le that learns to re-sample training set by meta- learning. Label Distribution Disentangling (LADE) 4 loss disentangles the source label distribution from the model prediction based on the optimal bound of the Donsker-Varadhan representation. Besides, Focal Loss (46) determines the weights for samples with the sample difficulty in the object detection task. and seltuence-balaneee Loss (IB) [3] assigns different weights to samples according to their influence on decision boundary. Our method can be naturally combined with many loss functions.
against class imbalance 45 5 saabel-Ditributtion-Nee
Other Methods Recently, two-stage algorithms have been proposed [ 5 24 such as MiSLAS
[27]. Mcanwhile, a Bilateral Branch Network (BBN) (25) unifies the representation and classifier learning stages to form a cumulative learning strategy. RoutIng Diverse Experts (RIDE) (28) uses multiple experts to reduce the variance and bias of the long-tailed classifier. In addition, some approaches employ meta-learning, such as MetaSAug (24) and BALMS (2). Causal Norm (CN) (29)
pinpoints the causal effect of momentum and extracts the unbiased direct impact of each instance.
Optimal Transport
a powerful tool, OT has been applied to generative models 47
48 49 50
5T], computer vision [52
56 57). text analysis [58 59 60 61). and etc. To the best of our knowledge, there are still very limited works for the imbalanced classification by means of OT. For example, (7) proposes re- weightiin method based on OT, and (62) introduces Optimal Transport via Linear Mapping (OTLM) to perform the post-hoc correction. Different from them, ours falls into the data augmentation group by mixing images. Another recent work is Optimal Transport for OverSampling (OTOS) (63), which moves random points from a prior uniform distribution into that of minority class samples based on OT. However, our proposed OTmix introduces OT to guide the mix between majority and minority images, which leverages the rich context of the majority classes guided by OT. Alignmixup (38] adopts OT to align two images and interpolate between two sets of features in a standard classification task, which is distinct from ours in terms of task and technical detail.
53 7
54 55
3 Background
Long-tailed Classification Given training set D= (Zi, vi) for multi-class problem with K classes, if each class & contains ny samples, we have that Σ ay = N. Without loss of generality, we can always assume n1 2 no 2 2 nk for the long-tailed problem. Denote
3

---

Cost Function
Image-mixing Process
P back
Bisc
Transport Plan
Classes
Sample
Baak
(Beed)
(Tai))
ore
Mixed samples
Class level
Majerto (fiead)
Mitartty (T44)
Clanse
Figure 2: Adaptive image-mixing method with optimal transport.
the model parameterized with A as g(0), which is trained on D with the well-known Empirical Risk Minimization (ERM) algorithm [2] ignores such class imbalance and performs poorly on minority classes 33
CutMix Here, we use the CutMix 37 to mix samples because of its simplicity and effectiveness, which replaces the image region with patch from another training image. Denote x E IR xxC and y denote a training image and its label, respectively. CutMix combines two training samples (zi4, 34) and (zj, Mj) and generates the new sample (Zij, Diy) as follows:
3i M 3 F + (1 M) @ k,, Øi = Ay + 1185-15-
where the combination ratio À is sampled from the beta distribution Beta(a, a)), @ is element-wise multiplication, and the binary mask M E {0, W> indicates where drop out and fill in from two images, 1 is a binary mask filled with ones. We provide more details about M in Appendix 7.1
Optimal Transport OT measures the minimal cost to transport between two probability distribu- We only provide brief introduction to OT for discrete distributions and more details. Denote two discrete probability distributions y= 1==aaa by b where both a and b are discrete probability vectors summing 1, m; and yj are the supports of the two distributions respectively, and 8 is Dirac function. Then the OT distance (T, C) where C E R200 Tell(p.s)
tions 35
64
65
66
67
refer the readers to 35
and y=
is formulated as follows: (T(p, p.. g) = min
is the cost matrix with
element Ci u1) which reflects the cost between &; and yy and the transport probability matrix E IR n is subject to [[ q) = (T Ei Ty The optimization problem above is often adapted to include popular entropic regularization term Eig Tij Tiy for
20
reducing the computational cost, denoted Sinkhorn algorithm (68]
Method
In this work, propose novel adaptive image-mixing data augmentation method based OT for long-tailed classification, where the overall framework is shown in Fig. 2 We consider distribution Phack over the background images mainly from majority classes and another distribution Q over the foreground images mainly from minority classes. By minimizing the OT distance between these two distributions, we design the sample-level and class-level cost functions and use the learned transport plan as guidance for the image- image-mixing process to generate more semantically meaningful mixed samples between the two distributions.
Main Objective Since majority classes usually have sufficient data and rich information, it is natural leverage the majority classes to enhance minority classes. In this work, we aim generate minority-centie images with majority contexts, combining the background and foreground images based on CutMix. The critical question is how pair a background image and foreground image. The first factor of the pairing strategy is that we would expect background image to be more likely from the majority classes and a foreground image to be more likely from the minority classes, which has been modeled in CMO 20 However, in addition to the first one, the second factor is the semantic similarity between the background and foreground images, which has not been studied in the literature and is the main focus of paper. Specifically, suppose set of candidate images

---

already satisfy the first factor, i.e., background and foreground images are from majority and minority classes, respectively. Pairing these images arbitrarily may not guarantee that the combined images are semantically meaningful and helpful, e.g., in Fig. T
In this paper, we aim to generate semantically meaningful images by proposing a pairing pro- cess that satisfies the first and second factors at the same time. Our proposal is derived from a distribution-matehhee perspective. First, we consider the distribution Pacck of the background images as the empirical distribution of all the images in the training dataset, which is a (discrete) uniform distribution:
Poack = Li=I
N N ((1,
Although Prack uniformly distributed, because the majority classes have significantly more images than the minority classes, it is natural that the samples from Prack are more likely to be images from the majority classes, which is what we expect. Secondly, we introduce the distribution over the foreground images, which is discrete distribution over all the training images:
Quere = Σ3 1 (23,
where wy is the weight of the j-th sample. In this case, we expect that a sample from Qiore is more likely to be an image of the minority classes. Thus, Qtore cannot be a uniform distribution. Accordingly,
specify Wy in several ways, such as the (smoothed) inverse class frequency [69] or the adequate number of samples [72]. Taking the inverse class frequency as the example,
70, 71
1/m where >> 0 controls we can sample the k-th class with the following probability: qu Σ Σk=1 1//g
the smoothness of qu. r 1 indicates the inverse class frequency and 1/2 means the smoothed version, i.e., if r increases, the weight of the minority class becomes increasingly larger than that of the majority class. Finally, we can use Wi
9k mi
if y = k.
For now, with construction of Piack and Qrare. the first factor is satisfied. For the second factor that requires the semantic similarity between a pair of background and foreground images, we formulate it as an optimization problem of the entropic OT:
OT (Ptack Qfore)
def
min
N.N EN CiJTas - [[E
Taj In Tt
(4)
where > is the hyper-parameter for the entropic constraint, Cij is the transport cost function, and the transport probability Tiy satisfies (T Qiore) == TTe E. Tip = wj. Ei = } Notably, as upper-bounded positive metric, Tıj indicates the transport probability between the i-th background image and the j-th foreground image, which can be naturally used to measure the importance of each background image for the foreground image when performing image- mixign.
Cost Function Cjj measures the distance between (æ,, Vi) in Pouck and (z), y3) in Qiare- As the main parameter for defining the transport distance between probability distributions, Ci. plays an important role in learning the optimal transport plan, which can be flexibly defined in different ways. For clarity, we can reformulate the concerned model g(0) as y2(9) (01); 02). where gi (0r) denotes the feature extractor parameterized with 0, and 92 (02) is the classifier parameterized with 02. We explore few conceptually intuitive options of Otj. A simple but straightforward way is defining the sample-level Ci with the features of x; and xy:
(5) where cos(..)) :) is the cosine similarity, 2, = 91 (z,;; 01) E IR and 2, - (1 01) E If denote the e-dimensional representation of z; and I,, respectively. Now Cig will be small if two samples have similar or close features, where the cost is influenced by the feature extractor 0z. Besides, we can also define the class-level Cij only using the label information. Specifically, normalized confusion matrix F E R.K
Cy =
2)),
R20 can be estimated with an unbiased validation set small balanced subset sampled from D. Summarizing the prediction results of the target model g, the confusion matrix F shows how the model g is confused when it makes predictions. Since we focus on combining the foreground images mainly from minority classes and background images mainly from minority classes, we can set the diagonal element in F to O, denoted as F, to avoid mixing the images from the same class. After that, we can define Cij with the ground-truth labels of two samples:
Cjj - F F
(6)
5

---

where F Fy.,, indicates the element of row yr, column yj, of the matrix F. If yı and yj are from the same class, Fy.., = and C25 1: if ys and V/ are not from the same class but are easy get to confused by the model F Fu.., would be large and Ci would be small. Now, Cay is affected by the feature extractor 01 and classifier 0z. Intuitively, based on the features and labels of samples, the cost function can also be defined as:
(7) where w is a whpeer-aaameeee parameter for balancing the feature and label information and Cas will be small if two samples have similar features and are from two confused classes but y: * Vz. Once the cost function Ciş is defined, it can be fed into (4) for learning the transport plan Ti between foreground images and background images, where a small Cij tends produce large Tij.
Ci w(1 cos(2i, 2z)) + (1 10--w)1
(11--10 (1 )
Image-mixing Process By minimizing the OT problem, the resultant transport plan provides an adaptive way to weigh the similarity between background and foreground images. Therefore, we can select the most suitable background image z.z' for the foreground image T3 as follows:
arg mrg = {i N j' J'EN
max Tiy JEN
(8)
We aim to generate new mixed samples by combining the foreground image 2; and the most relevant background image z. and their corresponding labels. As their name implies, Ty is used background image while x; provides the foreground patch, which can be pasted onto Ty. Recalling CutMix in ①I. the detailed mixing process is expressed as:
Biy = M x - (1 () C Ia. Diy XO/3 + +1 A)yi-
Training Details During the training process, we adopt a mini-batch setting to integrate our proposed OTmix with deep neural networks, where we learn T and @ alternatively. More specifically, at each iteration, we can sample a mini-batch Binre from Qore and another mini-batch Brack from Prack to build Qooe and Prack. both of which have M samples and M < N. In step (1), we can minimize the OT distance between Qiore and Prack with (4) to learn the transport plan T. In step (2), we can use the learned transport plan to select the most relevant background image x², for each foreground image x; based on (8). In step (3), we can generate the mixed samples with (9) and minimize the classification loss to train the model g parameterized by 0:
L (x
E
[E (yirg ((Tagg 0)1]
(10)
Since the cost function relies either the parameter 0 in the feature extractor or the parameter 0 in the whole model g. it might be inaccurate in the early training stage, resulting in an undesired transport plan matrix. Consequently, avoid selecting the unsatisfactory background image for the foreground image and generating harmful mixed samples, it is more beneficial to randomly select a'j
for in the early stage. To this end, adopt 3/andom ~ Bernoulli( f.) to decide whether randomly mixing the background and foreground pair, where T is the number of training epochs and t indicates the current epoch. In the early stage, such as t < 7/2, can get Vrandom =0 with high probability, where we randomly mix the background and foreground images like CMO [20]: in the late stage, such t > T/2, we more likely to sample Vrandem 1, where can learn T with (4) and select the most relevant background image for each foreground image. Moving beyond CMO, we introduce more adaptive and elegant mage-mixing data augmentation method for long- saile classification. We summarize our proposed method in Algorithm I and highlight steps (1), (2), and (3).
Using OTMix for Balanced Classifications In this case, our method can be viewed as better alternative to other augmentation methods that mix images randomly [37 [ 30). Applying OTMix in balanced classifications is similar to doing it in the imbalanced problems except that we construct both P Back and Qrore as uniform distributions.
Experiments
5.1 Experimental Settings and Implementation Details
Datasets Following 23 25
27
20
we evaluate our method on long-tailed classification benchmark
datasets: CIFAR-LT- 73 CIFAR-LT- 100 [73] ImageNet-LT (74). and iNaturalist 2018 (75) For

---

Algorithm Adaptive Image-mixing Method with Optimal Transport.
Require: Training dataset D, model with parameter 0, hyper-parameters (w, ax, B,e,r);
1: Build discrete distributions Quee with ( and Pauck with [
2: for ( 2... do
3: Sample mini-batch Blore from Quure and Brack from Poack; Build Quare with Biure and Prace with Brack
4: Sample Ymmdom Bermoull($);;
5: if V/unndo 0 then
6:
Generate the mixed samples with where we randomly choose 2y for dis
7:
else
8:
Step (I): Learn the transport plan T by minimizing O',, (QQaee Pbock) with 4
9:
Step (2): Select the most relevant background image x. for foreground image with B
Step (3): Generate the mixed samples (Tay', Meyr) with (
11: end if
12: Update g(++1) < g(t) - Vo with 10
13: end for
clarity, the imbalance factor is defined as the data point amount ratio between the most frequent and the least frequent classes, i.e., & nxK Among them, CIFAR-LT-10 (CIFAR-LT--00) are created Wh
from CIFAR-10 (CIFAR-100 76) with p= {100, 50, 10), respectively. ImageNet-LT, containing 1,000 classes with & 256, is created from ImageNet (74). Different from them, iNaturalist 2018 is real-world and large-scal scale imbalanced dataset with 8, 1,242 classes and 4 500.
Baseline Methods We consider several baseline methods: (1) Empirical risk minimization (ERM.
training the cross-entropy loss; (2) Mixup-baseds augmentation methods: Remix (2T). Unimix [2]
CMO 20): (3) Other augmentation methods: M2m (17), Open-sampling (18). FSA [23] SAFA 19
(4) Losses designed for long-tailed classifications: LD DRW (1). BALMS 22. IB [ LADE
4]: (6) Other methods: BBN (25). RIDE 28). Decouple (26). MisLAS 27
Implementation Details For all datasets, we use PyTorch (77] and SGD optimizer with momentum 0.9. Besides, we set w for defining cost function in (7) 0.05, for entropic constraint in as 0.01, r for smoothness in 3 as 1, and for sampling combination ratio as 4. For CIFAR-LT-I0 and CIFAR-LT-ION we use ResNet-32 [78] as the backbone following [ and use 240 epochs a single GTX 2080Ti and set the ini- tial learning rate 0.I, which is de- cayed by 0.1, 0.1, and 0.01 at the 100th, 160th, and 200th epochs. For the ImageNet-L.T, we employ ResNet- 50 the backbone following (20) and use 200 epochs on four GTX 2080Ti GPUs. The learning rate is initial- ized 0.1 and decays by 0.1, 0.1, and 0.01 at the 40th, 80th, and 160th epochs. For the iNaturalist 2018, we use ResNet-50 as the backbone fol- lowing [1] and train 210 epochs on four Tesla A A100 GPUs with an initial learning rate of 0.1, which decayed by 0.1 in the 30th, 80th, 130th, and 180th epochs. Performances are mainly reported as the overall top-1 errors (%). Following [79), also report the error rates three disjoint subsets: many-shot classes with more than 100 training samples, medium-sho classes with 20 to 100 samples, and few-shot classes with 20 samples.
Table 1: Top-1 emorr (%) ResNet-32 on CIFAR-LT-IO and CIFAR-LT- 100. ":": our reproduced results. "t": results reported in the original paper.
CIFAR-LTT-0
CIFAR-LT-HO0
100
10
30
10
ERM! LDAM ERM-DRW
26.9
22.9
12.9
63.2
56.3
43.4
24.3
18.9
11.9
58.0
54.3
41.8
LDAM-DRR
23.0
19.1
11.8
57.4
52.2
45.0
1B"
21.7
18.3
11.7
55.0
51.1
42.0
LADE
54.6
49.5
38.3
BBN
20.2
17.8
11.7
57.4
53.0
40.9
RIDE J Fipsts
28
18.1
16.0
13.7
51.4
48.6
40.2
MiSLAS!
14.3
10.0
53.0
47.7
36.8
ERM Open-sunling
18
22.4
18.2
10.6
59.7
55.2
41.9
FRM M2m
17
21.7
12.1
57.1
41.8
LDAM-DRW SAFA
ERM Remix
24.6
11.8
58.1
40.6
ERM UniMix
23.5
58.5
FRM
18.6
11.51
56.1
$1.7"
40.5
RIDE experts CMO 200
17.87
15.4
12.17
50.07
47.0"
39.8
ERM - OTmix
21.7
16.6
9.8
53.6
49.3
38.4
LDAM OTmix
22.3
18.0
12.0
56.3
50.9
41.5
DRW OThixx
16.9
13.8
9.4
52.0
47.4
37.3
L.DM--DRW - OTmis
18.2
16.0
11.8
52.0
47.6
41.0
BALMS OTmix
16.0
13.5
9.8
53.2
17.7
37.7
experts) OTmix
7

---

Table Top-1 errors (%) ResNet-50 on ImageNet-LT and iNaturalist 2018 "*": results reported in CMO. "f": results reported in origin paper.
IntageNet-LT
iNaturalist
Many
Mediom
Few
Many
Medium
ERM 20
58.4
36.0
66.2
94.2
39.0
26.1
36.5
44.5
52.7*
71.2*
36.31
LDAM-DRW
50.2
39.6
53.1
69.3
30.0
30.0
29.8
30.1
BALMS
49.0
39.1
51.2
67.9
30.0
30.0
29.8
30.1
IB
34,6
LADE
48.1
37.7
50.7
68.8
30,0
Decouple-cRT
52.7
41.2
73.9
31.8
26.8
33.9
Decouple LWS WS
26
52.3
42.9
$4.8
70.7
30.5
29.0
30.2
31.8
BBN
30.4
MISLAS
47.3
28.4
26.8
27.6
29.6
RIDE (3 expens)
28
45.1
33.8
48.3
65.1
27.8
29.8
27.8
27.3
experts)
63.5
29.1
29.1
FRM M2m
17
57.8
Focal Loss+ FSA
23
34.1
LDAM-DRS SAPA
19
30.2
ERM Remix
21
58.3
38.7
ERM + CMO
50.9
33.0
$7.7
79.5
31.1
23.1
30.7
33.4
RIDE experts) CMO
20
43.8
33.6
46.1
64.4
27.2
31,3
27.4
26.9
ERM + OTmix
48.0
30.0
54.1
71.7
30.5
30.7
29.5
31.6
DRW + OTmix
46.6
33.0
51.0
69.6
28.9
29.4
28.1
29.6
31.5
BALMS- OTmixx
44.4
36.0
47.6
57.3
28.5
28.9
28.0
29.2
RIDE (3 exports) OTmix
42.7
40.6
43.5
55.9
27.0
28.7
27.2
26.2
5.2
Experiments Long-tailed Classification
We conduct experiments on CIFAR-LT-10 and CIFAR-LT-100 of ResNet-32 in Table I where Ommi++RRM achieves better performance than the mixup- based augmentation methods with ERM loss, especially for CMO+ERM. It confirms the validity of selecting suitable pair to mix. Besides, OTmix with ERM loss produces comparable performance to complex long-tailed classification methods, such specially designed losses and other augmentation methods. Considering OTmix can also be combined with some existing long-tailed losses and model architectures, we further combine it with LDAM-DRW loss (I. BALMS loss (2), and also the network with multiple experts (RIDE) [28). clear that OTmix equipped with BALMS loss, DRW loss, and RIDE outperforms most of the competing methods except CIFAR-LT-I0 with & 10, where it is only weaker than MisLAS. These results illustrate the superiority of the proposed OTmix.
We further perform experiments on large-scale imbalanced ImageNet-LT and iNaturalist 2018 datasets. Table [ demonstrates that OTmix based on ERM loss can still produce a comparable performance when compared with competing long-tailed losses, related based, and other data augmentation methods. Similarly, combining ours with other losses or model architecture can overall outperform these related baselines on ImageNet-LTT and iNaturalist 2018, where RIDE (3 experts) + OTmix performs best. Furthermore, our method can improve the performance of the medium and few classes, which is the goal of long-tailed methods, We compare OTmix with mixup-based augmentation methods under various loss functions in Appendix 7.2 and computational cost Appendix 7.4
5.3 Experiments Balanced Classification
To validate whether our proposed OTmix can be used for the balanced classification task, we adopt the
Table Top-1 errors (%) of our methods using ResNet-32 on balanced datasets.
commontis commonly-used CIFAR-10 (76) and CIFAR-100 (76).
Method
CHEAR-10
CIFAR-100
Notably, our proposed OTmix can be flexibly combined with the classical mixup-based methods when mixing images, where we consider Mixup (32). Cutmix (37].
ResNet-32 (baseline)
11.8
34.2
-Mixup
-Tmix (iixsp))
8.1
32.3
and SaliencyMix [30]. We train the networks using the same details as on the CIFAR-LT. At each training iteration, we randomly sample two bini-bacches, i.c.,
CClmix 37
8.6
31.6
OTTix (Cutmix)
5.8
28.0
SaliencyMix
30
7.2
31.2
-TTix (SaliencyMix)
Biere and Boack from the balanced training dataset. Table 3 CIFAR datasets, where OTmix produces better performance. indicates the benefit of building better image pair to perform image-mixx mage-mixxng data augmentation for the balanced classification task. Moreover, OTmix (Saliency Mix) achieves the best top-1 error of 5. % and 21.7% on CIFAR-10 and CIFAR-100 100 datasets, respectively. These results show that OTmix is still effective balanced
lists the experimental results on balanced

---

classification tasks and can be combined with some advanced mixup-based methods, proving its flexibility and availability.
5.4 Analytical Experiments
Here we implement the ablation study under various settings on CIFAR-LT-I00 with y= 100 and summarize the results in Table Our best- pert-perfonnnn method uses the combined cost function with cosine similarity, sets the diagonal element in the confusion matrix F as O, and relies on the maximum value in the transport plan matrix to select the background image for each foreground image. To reveal the influence of the cost function, we denote the cost function by only using the sample-level cost and the class-level cost as "'Sapplelevel cost only" and "Class-level cost only", respectively Besides, we also consider using euclidean distance rather than cosine similarity to define the cost function called "Euclidean distance". Compared with ours, which combines the sample-level and class-level costs and uses cosine similarity, all these variants perform slightly worse, demonstrating the benefit of combining features and labels to compute the cost function and the effectiveness of cosine similarity.
To avoid mixing the background and foreground images from the same category, we set the diagonal element in F as and use the resultant F to compute the cost function. Here, we consider its variant by calculating the cost function using the original confusion matrix F, denoted as "Keeping same category", which can be found to outperform by OTmix. It shows that avoiding mixing samples from the same class is beneficial, no matter from minority or majority classes. Besides, we adopt the multinomial distribution, denoted as "Probability selection"' select the corresponding background image for the i-th foreground image, whose probability vector can be formulated as p coftmax(TT..:.11 ). We observe that selecting background images according to probability distribution is inferior to using the maximum value, where the latter chooses the most relevant background image for each foreground image mainly from minority classes without randomness. proves the necessity of using the maximum value in the transport plan to build pair for mixing.
Table 4: Ablation study on CIFAR-L.T 100.
Method
All
Many
Medium
Few
ERM + OTmix
S3.6
27.7
53.2
83.3
Sample-lvel cost only
55.1
30.8
56.1
85.5
Class-level oost only
54.5
30.1
55.9
84.5
Euclidean distance
54.1
27.9
54.0
84.8
Keeping same category
54.9
29.0
55.1
85.0
Probability selection
55.0 28.0
55.7
85.6
Confusion matrix
57.1
29.4
59.5
86.8
Furthermore, we use the confusion matrix itself ("Confusion matrix" in Table 4 design the image- mixing pair instead of OT. The confusion matrix can be from a pre-trained model on the imbalanced training set with standard ERM loss. Specifically, after sampling the background and foreground images, if the class 11 of the foreground image is most easily confused with 22 of the background image (k₃ * k:2), we use them to construct a pair. However, using the confusion matrix to build the image-mixing pair is inferior to ours. The possible reason is that OTmix provides more adaptive way to measure the importance of each background image for the foreground image at each training iteration. We provide more analytical results in Appendix 7.3
5.5
The Learned Transport Plan for Image-mixing
Recall that our method adaptively learns transport plan be- tween a batch of foreground images and a batch of background images. Fig. 3 shows
hosse
0.07 0.22
0.10
0.07
0.05
0.12
0.06
0.16
0.08
0.08
dua
learned transport plan on CIFAR- LT-10 with o = 100, whose list of class names is fairplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck) from k = 1 to 10. To facilitate the visualization, we set the size of Brore and Brack as M 10, respectively. We observe that the learned transport plan can effectively capture the seman- tic correlations between foreground and background images. For example, the foreground image from the "truck" class (from the "ship" class) is highly related the background image from the "automobile" class (from the "airplane" class). These correlations align with the confusion matrix in Fig. 6 of Appendix 77. however, OTmix is adaptive and specific to batches, while the confusion matrix is global and fixed. In addition to capturing the class-level similarity information, we can find that different background images from the same class have different importance to the same foreground image, i.e., the sample-level similarity information. For instance, the transport
Foreground labels
dhop
2.00001 0.15
0.24
0.14
0.02..001
0.00
0.18 0.07
0.
mmg
0.03 0.24 0.03
0.62
0.05
0.05
0.09
0.35
0.02 0.13
auttomebile frur
0.01
0.00
0.11 0.03 04
desr
0.02 0:53
0.02
0.02
0.04
0.35
0.03 0.13
bird
0.03
0.06
0.03
0.05 0.0 0.18
0.01
0.0 0.00 0.00 0.15 0.11 0.15
Wishone
lidd
Background labels
Figure 3: Our learned transport plan on CIFAR-LT-10 with p= 100.
9

---

probabilities between the foreground dog image and two background cat images are 0.75 and 0.24, respectively. We attribute that our designed cost function based on the sample-level feature and class-level confusion matrix when learning the transport plan. Therefore, learned transport plan is an effective and adaptive way to guide the image-mixing process for long-tailed classification. We further compare the image-mixing statistical results of OTmix and CMO in Appendix [7. at one epoch, i.e., the probability each class in foreground distribution being mixed with each class in background distribution.
5.6 Visualization Results and Analysis
T-SNE Visualization To gain additional insight, we provide the T-SNE visualization of BALMS, BALMS+CMO, and BALMS+OTmix CIFAR-LT-I0 with yp 100 in Fig. 4 Compared with BALMS+CMO, the majority class more separable from the minority class2 in OTmix. Besides, the samples within the same class in proposed method usually closer than in CMO, i.c., small intra-class. It indicates that our proposed method can generate more effective mixed samples to improve the long-tailed classification performance.
Moncly 2
(a) BALMS
(b) BALMS
(c) BALMS+OTmix
Figure 4: T-SNE visualization of BALMS, BALMS+CMO, and BALMS+OTmix CIFAR-LT-IO.
Mixed Image Pairs Visualization To visually witness the intricate dynamics of the mixing process in our method, we provide the visualization results image pairs for semantically meaningful mixed images ImageNet-LT. As shown in Fig. [] we plot the more semantically reasonable and relevant background image for each foreground image given the learned transport plan. For example, the semantic information of "Grey fox" is semantically relevant to "Dhole". It visually reveals that our method can generate meaningful and advantageous samples, benefiting the balanced and imbalanced classification tasks. More visualization results available in Appendix 7.5
Foregrount
Linnet
Spaniel
Admiral
Lifebout
Lemon
Expresso
Drake
Ski
Grey fox
Background
Junco
Persian cat
Fly
Liner
Granny Smith
Car
Cup
Black Swan
Snowmobile
Dhole
Mixed
Figure 5: The visualization results of image pairs for meaningful mixed images ImageNet-LT.
Conclusion
We have introduced novel adaptive image-mixing augmentation method based on OT for long- tailed classification, with the goal of generating more semantically meaningful samples. We view the background images mainly from majority classes and the foreground images mostly from minority classes as two distributions, where we aim to minimize the OT distance between these two distributions. Moreover, we further design a combined cost function based on sample-level and label-level information. By viewing the learned transport plan as guidance to build a pair for image-mixing, we provide an effective way to weigh the background image for each foreground image. Our proposed OTmix has shown appealing properties that can be either combined with existing long-tailed methods or applied to balanced classifications. Experimental results validate that OTmix achieves competing performance commonly long-tailed tailed problems and balanced datasets.
10

---

References
[1] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. "Learning Imbalanced Datasets with Label-Distribution-Aaes
Margin Loss". Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). 2019.
[2] Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. "Balanced meta-softmax for long-tailed visual recognition". Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). 2020.
[3] Seulki Park, Jongin Lim, Younghan Jeon, and Jin Young Choi. "Inftuence-balanced loss for imbalanced visual classification". Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021, pp. 735-744.
[4] Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, et al. "Disentangling label distribution for long-tailed visual recognition". Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021, pp. 6626-6636.
[5] Jun Shu, Qi Xie. Lixuan Yi, Qian Zhao, et al. "Meta-wetd seeta-weight-eee net: Learning an explicit mapping for sample weighting". Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). 2019.
[6] Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. "Learning to reweight examples for robust deep learning". International conference on machine learning. 2018, pp. 4334-4343,
[7] Dandan Guo. Zhuo Li, Meixi Zheng, He Zhao, et al. "Learning to Re-weight Examples with Optimal Transport for Imbalanced Classification". Advances in Neural Information Processing Systems. 2022. pp. 25-21-225555
[8] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, et al. "Exploring the limits of weakly supervised pretraining". Proceedings of the European conference on computer vision (ECCV). 2018, pp. 181-196.
[9] Jialian Wu, Liangchen Song. Qian Zhang, Ming Yang, and Junsong Yuan. "Forestdet; Large- vocabulary long-tailed object detection and instance segmentation", IEEE Transactions on Multimedia 24 (2021), pp. 3693-3705.
[10] Xinting Hu, Yi Jiang, Kaihua Tang, Jingyuan Chen, et al. "Learning to segment the tail". Proceedings of the IEEF/CVF conference on computer vision and pattern recognition. 2020. [11] Junran Peng, Xingyuan Bu, Ming Sun, Zhaoxiang Zhang, et al. "Large-scale object detection in the wild from imbalanced multi-labels" Proceedings of the IEEE/CVF conference computer vision and pattern recognition. 2020, pp. 9709-9718.
[12] Nitesh Chawla, Kevin W Bowyer, Lawrence Hall, and W Philip Kegelmeyer. "SMOTE: synthetic minority over-sampling technique". Journal of artificial intelligence research 16 (2002), pp. 321-357.
[13] Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. "A systematic study of the class imbalance problem in convolutional neural networks". Neural networks (2018), pp. 249-259.
[14] Guo Haixiang, Li Yijing, Jennifer Shang, Gu Mingyun, al. "Learning from class-imbalanced data: Review of methods and applications". Expert systems with applications (2017), pp. 220- 239.
[15] Debashree Devi, Biswajit Purkayastha, et al. "Redundancy-driven modified Tomek-link based undersampling: A solution to class imbalance". Pattern Recognition Letters (2017), pp. 3-12.
[16] Haibo He and Edwardo A Garcia. "Learning from imbalanced data". IEEE Transactions knowledge and data engineering (2009), pp. 1263-1284.
[17] Jachyung Kim, Jongheon Jeong, and Jinwoo Shin. "M2m: Imbalanced classification via major-to-minor translation". Proceedings of the IEEE/CVF conference computer vision and pattern recognition. 2020. pp. 13896-13905.
[18] Hongxin Wei, Lue Tao, Renchunzi Xie, Lei Feng, and Bo An. "Open-sampling: Exploring out-of-distribution data for re-balancing long-tailed datasets". International Conference on Machine Learning. 2022, pp. 23615-23630.
[19] Yan Hong, Jianfu Zhang, Zhongyi Sun, and Ke Yan. "Safa: Sample-adaptive feature aug- mentation for tailed image classification". European Conference on Computer Vision.
2022.
[20] Seulki Park, Youngkyu Hong, Byeongho Heo, Sangdoo Yun, and Jin Young Choi. "The major- ity can help the minority: Context-rich minority oversampling for long-tailed classification". Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
11

---

[21] Hsin-Ping Chou, Shih-Chieh Chang, Jia- Yu Pan, Wei Wei, and Da-Cheng Juan. "Remix: rebalanced mixup". Proceedings of the European conference on computer vision (ECCV). 2020, pp. 95-110.
[22] Zhengzhuo Xu, Zenghao Chai, and Chun Yuan. "Towards calibrated model for long- -ailed visual recognition from prior perspective". Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). 2021, pp. 7139-7152.
[23] Peng Chu, Xiao Bian, Shaopeng Liu, and Haibin Ling. "Feature space augmentation for long-tailed data". European Conference on Computer Vision. 2020, pp. 694-710.
[24] Shuang Li, Kaixiong Gong, Chi Harold Liu, Yulin Wang, et al. "Metasaug: Meta semantic augmentation for long-tailed visual recognition". Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021. pp. 5212-5221.
[25] Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. "Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition". Proceedings of the IEEE/CVF conference on compuler vision and pattern recognition. 2020, pp. 9719-9728.
[26] Bingyi Kang. Saining Xie, Marcus Rohrbach, Zhicheng Yan, et al. "Decoupling representation and classifier for long-tailed recognition". Eighth International Conference on Learning Representations (ICLR). 2020.
[27] Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. "Improving calibration for long-tailed recognition". Proceedings of the IEEF/CVE conference on computer vision and pattern recognition. 2021, pp- 16489-16498.
[28] Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella Yu. "Long-tailed Recognition by Routing Diverse Distribution-Awe -ware Experts". International Conference on Learning Representations: 2021.
[29] Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. "Long-Taileds Classification by Keeping the Good and Removing the Bad Momentum Causal Effect". Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). 2020.
[30] AFM Shahab Uddin, Sirazam Monira, Wheemyung Shin, TacChoong Chung, Sung-Ho Bae, et al. "SaliencyMix: A Saliency Guided Data Augmentation Strategy for Better Regularization". International Conference on Learning Representaatonss 2021.
[31] Jie-Neng Chen, Shuyang Sun, Ju He, Philip HS Torr, et al. "Transmix: Attend to mix for vision transformers". Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022. pp. 12135-12144.
[32] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. "mixup: Beyond Empirical Risk Minimization". International Conference on Learning Representations. 2018.
[33] Zongbo Han, Zhipeng Liang, Fan Yang, Liu I. Liu, al. "Umix: Improving importance weighting for subpopulation shift via uncertainty- aware mixup". Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). 2022.
[34] Adrian Galdran, Gustavo Carneiro, and Miguel A González Ballester. "Balanced -Balanced-mixup for highly imbalanced medical image classification". Medical Image Computing and Computer Assisted Intervention. 2021, pp. 323-333.
[35] Gabriel Peyré and Marco Cuturi. "Computational optimal transport". Foundations and Trends in Machine Learning (2019), pp. 355-607.
[36] Lu Yang, He Jiang, Qing Song, and Jun Guo. "A survey on long-tailed visual recognition". International Journal of Computer Vision (2022), pp. 1837-1872.
[37] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, et al. "Cutmix: Regularization strategy to train strong classifiers with localizable features". Proceedingy of the IEEF/CVF international conference on computer vision. 2019, pp. 6023-6032.
[38] Shashanka Venkataramanan, Ewa Kijak, Laurent Amsaleg, and Yannis Avrithis. " "Alignmixup: Improving representations by interpolating aligned features". Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. 2022, pp. 19174-19183.
[39] Jihao Liu, Boxiao Liu, Hang Zhou, Hongsheng Li, and Yu Liu. "Tokenmix: Rethinking image mixing for data augmentation in vision transformers". European Conference on Computer Vision. 2022, pp. 455-471.
[40] Lingfeng Yang, Xiang Li, Borui Zhao, Renjie Song, and Jian Yang. "Recursivemix: Mixed learning with history",. Proceedings of the Advances Neural Information Processing Systems (NeurlPS). 2022.
12

---

[41] Zhi Hou, Baosheng Yu, and Dacheng Tao. "Batchformer: Learning to explore sample rela- tionships for robust representation learning". Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, pp. 7256-7266.
[42] Anh Tuan Bui, Trung Le, Quan Hung Tran, He Zhao, and Dinh Phung. "A Unified Wasserstein Distributional Robustness Framework for Adversarial Training". International Conference on Learning Representations. 2021.
[43] Irena Gao, Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. "Out-of- Domain Robustness via Targeted Augmentations". International Conference on Machine Learning. 2023.
[44] Sara Beery, Yang Liu, Dan Morris, Jim Piavis, et al. "Synthetic examples improve general- ization for rare classes". Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision 2020.
[45] Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. "Class-balanced loss based on effective number of samples". Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019, pp.
[46] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. "Focal loss for dense object detection". Proceedings of the IEEE international conference on computer vision. 2017, pp. 2980-2998
[47] Aude Genevay, Gabriel Peyré, and Marco Cuturi. "Learning generative models with sinkhorn divergences". International 1617.
Inveraanes,.Interrattoonn Conference on Artificial Intelligence and Statistics. 2018, pp. 1608-
[48] Giorgio Patrini, Rianne Van den Berg, Patrick Forre, Marcello Carioni, et al. "Sinkhorn autoencoders". Uncertainly in Artificial Intelligence. 2020, pp. 733-743.
[49] Viet Huynh, Dinh Phung, and He Zhao. "Optimal transport for deep generative models: state of the art and research challenges". International Joint Conference on Artificial Intelligence
2021. 2021, pp.
[50] Tung- Long Vuong. Trung Le, He Zhao, Chuanxia Zheng, al. "Vector Quantized Wasserstein Auto-Encodes International Conference on Machine Learning. 2023.
[51] Vy Vo, Trung Le, Long-Tung Vuong, He Zhao, et al. "Learning Directed Graphical Models with Optimal Transport". arXiv preprini arXiv:2305. 15927 (2023).
[52] Tuan Nguyen, Trung Le, He Zhao, Quan Hung Tran, et al. "Most: Multi-source domain adap- tation via optimal transport for ttudent-teacher learning". Uncertainty in Artificial Intelligence. 2021, pp. 225-235.
[53] Dan dan Guo, Long Tian, Minghe Zhang, Mingyuan Zhou, and Hongyuan Zha. "Learn- ing prototype-orienter set representations for meta-learning". International Conference on Learning Reprexentations. 2022.
[54] Dandan Guo, Long Tian, He Zhao, Mingyuan Zhou, and Hongyuan Zha. "Adaptive Distribution Calibration for Few-Shot Learning with Hierarchical Optimal Transport". Advances in Neural Information Processing Systems. 2022, pp. 6996-7010.
[55] Long Tian, Jingyi Feng, Xiaoqiang Chai, Wenchao Chen, al. "Prototypes-oriented Transduc- tive Few-shot Learning with Conditional Transport". Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision. 2023.
[56] Miaoge Li, Dongsheng Wang, Xinyang Liu, Zequn Zeng, et al. "PatchCT: Aligning Patch Set and Label Set with Conditional Transport for Multi-Label Image Classification"" IEEE/CVF International Conference on Computer Vision. 2023, pp. 15348-15358.
[57] Dongsheng Wang, Miaoge Li, Xinyang Liu, MingSheng Xu, et al. "Tuning Multi-mode Token- level Prompt Alignment across Modalities". Advances Neural Information Processing Systems (2023).
[58] Viet Huynh, He Zhao, and Dinh Phung. "Otlda: A geometry-aware optimal transport approach for topie modeling". Advances in Neural Information Processing Systems 33 (2020), pp. 18573-
18582.
[59] He Zhao, Dinh Phung, Viet Huynh, Trung Le, and Wray Buntine. "Neural Topic Model via Optimal Transport". International Conference on Learning Representations. 2021. [60] Dongsheng Wang, Dandan Guo, He Zhao, Huangjie Zheng, et al. "Representing Mixtures of Word Embeddings with Mixtures of Topic Embeddings". International Conference on Learning Representations. 2022.
13

---

[61] Dongsheng Wang, He Zhao, Dan Dan Guo, Xinyang Liu, et al. "BAT-Chain: Bawes Transport Chain for Topic Hierarchies Discovery" (2022).
[62] Hanyu Peng, Mingming Sun, and Ping Li. "Optimal transport for long-tailed recognition with learnable cost matrix". International Conference on Learning Representations. 2022.
[63] Yuguang Yan, Mingkui Tan, Yanwu Xu, Jiezhang Cao, et al. "Oversampling for imbalanced data via optimal transport". Proceedings of the AAAI Conference on Artificial Intelligence. 2019.
[64]
Jean-David Benamou, Guillaume Carlier, Marco Cuturi, Luca Nenna, and Gabriel Peyré "Iterative Bregman projections for regularized transportation problems". SIAM Journal on Scientific Computing (2015), A1111-A1138.
[65] Lenaic Chizat, Gabriel Peyré, Bernhard Schmitzer, and Prançois-Xavies Vialard. "Scaling algorithms for unbalanced optimal transport problems". Mathematics of Computation (2018),
pp. 2563-2609.
[66] R Flamary, N Courty, D Tuia, and A Rakotomamonjy. "Optimal transport for domain adapta- tion". IEEE Trans. Pattern Anal. Mach. Intell (2017). pp. 1853-1865.
[67] Aude Genevay, Marco Cuturi, Gabriel Peyré, and Francis Bach. "Stochastic optimization for large-scale optimal transport". Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). 2016.
[68] Marco Cuturi. "Sinkhorn distances: Lightspeed computation of optimal transport". Proceeddioss of the Advances Neural Information Processing Systems (NeurlPS). 2013.
[69] Chen Huang. Yining Li, Chen Change Loy, and Xiaoou Tang. "Learning deep representation for imbalanced classification". Proceedings of the IEEE conference computer vision and pattern recognition. 2016, pp. 5375-5384.
[70] Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. "Learning to model the tail". Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). 2017.
[71] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, et "Exploring the limits of weakly supervised pretraining". Proceedings of the European conference on computer vision (ECCV). 2018, pp. 181-196.
[72] Yin Cui, Menglin Jia, Tsung-Yi Yi Lin, Yang Song, and Serge Belongie. (Class-balanced loss based effective number of samples". Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019, pp. 9268-9277
[73] Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. "Class- balanseb loss based on effective number of samples". Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019, pp. 9268-9277.
[74] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, et al. "Imagenet: A large-scale hierarchical image database". IEEE conference on computer vision and pattern recognition. 2009.
[75] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, et al. "The inaturalist species classification and detection dataset".. Proceedings the IEEE conference on computer vision and pattern recognition. 2018, pp. 8769-8778.
[76] Alex Krizhevsky, Geoffrey Hinton, et al. "Learning multiple layers of features from tiny images". Handbook of Systemic Autoimmune Diseases 1.4 (2009).
[77] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, et al. "Pytorch: An imperative style, -eghhperformeece deep learning library". Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). 2019.
[78] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. "Deep residual learning for image recognition". Proceedings of the IEEE conference on computer vision and pattern recognition. 2016, pp. 770-778.
[79] Ziwei Liu, Zhonggi Miao, Xiaohang Zhan, Jiayu Wang, al. "Large-scale long-tail tailed recognition in an open world". Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019, pp. 2537-2546.
14

---

7 Appendix
7.1 Details CutMix
To sample the binary mask M, first sample the bounding box coordinates B (Tas Ta Two Th) indicating the cropping regions on z; and z,. The region B in æ; is removed and filled with the patch cropped from B of z.. In our experiments, we sample rectangular masks M whose aspect ratio proportional to the original image. The box coordinates are uniformly sampled according to the:
Tx Unif(0, W),
WvI- Ty Uni H), Th =
(11)
making the cropped area ratio {0, 1) 1) is decided by filling with 0 within the bounding box B, otherwise 1.
TuTh 1 With the cropping region, the binary mask M WH
7.2 More Comparison Results
In this section, we conduct comparison studies with mixup-based long-tailed methods, including Remix [11] Unimix [22], and CMO [20] under various loss functions. We summarize the results on and CIFAR-LT-100 in Table 6 list the results on ImageNet-LT and iNaturalist 2018 in Table We can find that our OTmix surpasses related mixup- mixup-based methods with varying loss functions and enhances the imbalanced classification. Besides, OTmix with different losses produces different classification performances due to their characteristics, where OTmix with BALMS can achieve better or competing performance than OTmix with other loss functions. These results reveal the effectiveness of our proposed method when combined with other loss functions.
Table 5: Top-1 errors (%) of mixup-based long-tailed methods under various loss functions on ImageNet-LT and iNaturalist 2018. "*": results reported in CMO. "f": results reported in origin paper.
Method
ImageNet-IT
iNaturalist 2018
Mixup
Loss
ALL
Many
Medium
Few
ALL
Many
Medium
Few
None
ERM
58.4
36.0
66.2
94.2
39.0
26.1
36.5
44.5
Remixt
ERM
58.3
38.7
CMO
ERM
50.9
33.0
57.7
79.5
31.1
23.1
30.7
33.4
OTmix
ERM
48.0
30.0
54.1
77.7
30.5
30.7
29.5
31.6
None
ERM-DRW
49.9*
38.3*
52.7*
71.2*
36.3
Remixt
ERM-DRW
29.5
CMO
ERM-DRW
48.6
39.2
51,4
64.5
29.1
31.8
29.8
27.8
OTmix
ERM-DRW
46.6
33.0
51.0
69.6
28.9
29.4
28.1
29.6
None
LDAM-DRW
50.2
39.6
53.1
69.3
30.0
30.0
29.8
30.1
CMO
LDAM-DRW
48.9
38.0
52.6
69.2
30.9
24.7
30.5
32.7
OTmix
LDAM-DRW
47.5
37.2
48.9
71.8
30.4
31.5
29.8
30.6
None
BALMS
49.0
39.1
51.2
67.9
30.0
30.0
29.8
30.1
CMO
BALMS
47.7
38.0
50.9
63.3
29.1
31.2
30.0
27.7
OTmix
BALMS
44.4
36.0
47.6
57.3
28.5
28.9
28.0
29.2
7.3 More Analytical Results
Confusion Matrix To verify whether our method improves the performance of minority classes, we show the confusion matrices of ERM-DRW, CIFAR-LT-I0 with & = 100 Fig. 6 We can find that ERM-DRW suffers a severe performance drop in the minority classes even though it can almost accurately predict the samples in the majority classes. FRM-DRW+
and ERM-DRW+OThe on
can improve the accuracy of the minority classes, which coincides with the statement of CMO. OTmix further enhances the generalization of minority classes and

---

Table Top-1 (%) errors of mixup-based long-tailed methods with various loss functions on CIFAR- LT-10 and CIFAR-LI -000 "# " our reproduced results. "f": results reported in the original paper.
Method
CIFAR-LT-10
CIFAR-LT-ION
Mixup
Loss
100
50
10
100
50
10
None#
ERM
26.9
22.9
12.9
63.2
56.3
43.4
CMO
ERM
25.0F
18.6
11.5
56.1
51.7
40.5
Remix
ERM
24.6
11.8
58.1
40.6
UniMix
ERM
23.5
58.5
OTmix
ERM
21.7
16.6
9.8
53.6
49.3
38.4
None
LDAM
26.3
21.6
13.4
61.1
56.0
44.4
CMO
LDAM
25.9
22.7
13.1
58.0
54.3
44.1
UniMix
LDAM
24.6
58.3
OTmix
LDAM
22.3
18.0
12.0
56.3
50.9
41.5
None
ERM-DRW
24.3
18.9
11.9
58.0
54.3
41.8
Remixt
ERM-DRW
20.2
11.0
53.2
38.8
CMO
ERM-DRW
19.5#
16.6*
11.3
53.01
49.1
38.31
OTmix
ERM-DRW
16.9
13.8
9.4
52.0
47.4
37.3
None
LDAM-DRW
23.0
19.1
11.8
57.4
52.2
45.0
Remix
LDAM-DRY
20.7
13.2
55.0
40.5
CMO
LDAM-DRW
19.0
16.2
12.4
52.8
48.3
41.6t
OTmix
LDAM-DRW
18.2
16.0
11.8
52.0
47.6
41.0
None
BALMS
22.7
19.1
11.8
58.0
53.1
41.6
CMO
BALMS
19.7
15.9
11.0
53.4
48.6
37.71
OTmix
BALMS
16.0
13.5
9.8
53.2
47.7
37.7
None
Focal
30.4
23.4
13.6
61.6
56.3
45.0
CMO
Focal
29.0
22.6
12.5
58.1
53.6
41.8
OTmix
Focal
27.0
20.9
12.0
57.8
53.3
40.6
None
CB Softmax
26.0
21.1
12.3
CMO
CB Softmax
25.7
20.2
12.2
OTmix
CB Softmax
24.0
18.8
12.0
None
CB Sigmoid
26.5
22.4
12.5
CMO
Sigmoid
27.4
20.8
12.1
OTmix
CB Sigmoid
24.9
20.6
11.8
maintains performance in majority classes, which thus outperforms the strong baselines on the overall performance. In Fig we plot the classification results for each class on CIFAR-LT-10 with = 100, where we adopt the BALMS loss and LDAM loss, respectively. Compared with these baselines. OTmix provides significant improvement in minority classes. We specifically note that the proposed method improves the accuracy over BALMS by 23% class 9 while degrading the accuracy for class 0 and class by less than 2%. These results indicate that ours can achieve more balanced classifier and ameliorate the generalization of minority classes.
over L.AAM by 15% for the least frequent
Discussion of Hyper-parameteer and Training Settings To analyze the effect of different hyper- parameters and settings of OTmix, we conduct analytical experiments on CIFAR-LT- with yp 100. The hyper-yaranee parameters include w, a, and r, respectively. w (7) is employed to manage the degree of combination of the confusion matrix and feature information in Fig. B(a. The best performance is achieved when w 0.05, indicating the class-level cost based on confusion matrix dominating the cost function. a affects the combined ratio in Cutmix in Appendix 77. As shown in Fig. b)) the larger the value of is, the more complementary the distribution tends to closer uniform distribution. The OTmix achieves accuracy of 46.4% when 4.

---

ERM-DRW
ERM-DRW CMO
ERM-DRI DRV OTmiX
0:00
B.02
3/22 3:13 DLOX
0.00 0.99
0.000.00
0.00
2.00
0.20 0.01
0.00 3.00
5:0
2:00
0:00 0.00
0.5
True label
6.04 0.000.0.770
0.0
0.06
0.07
3.01
0.00 0.00
02 0.01600 6.04
0.03
0.08
0.02 0.0
02
9.00 3/44
0,3 0:33
0.00
0.5
0.04 0.00 05 0.19
0.00 0.00
23
0.00
0.04
0.170.0.
8:71
0.01
0.02
0.00
0.00
0.01
0.00
902
9.12
9:63
0.0.0.000.0000000000
0.4
0.000.07000
6.400
0:44
0.000000000000000000000000000000
5.07 0.03
4..00
0.00
0.03
0.06
0.10
0.0000000000
0.00
0.72
0.00 0.00
0.03
0.02
9.04
0.000.00000
0.02 0.00 2.00
0344
0.03
0.12
0.2
5.23 0.00
0:31
0.000
0.000000000000000000000000000000
8:01
Predicted
Predicted label
Predicted label
Figure 6: Confusion matrices of the ERM-DRW, FRM-DRW+CMG and ERM-DRW+OTmis on CIFAR-LT-10 with & = 100.
True label
Tuue 1830
40
50
60
70
100
80
70
80
90
100
Accuracy (%)
Accuracy (%)
(a) BALMS baseline
(b) LDAM baseline
Figure The classification results of different methods for each class on CIFAR-LT-IO with وP 100, where (a) uses the BALMS loss and (b) adopts the LDAM loss. Class 0 stands for the majority class, and class 9 stands for the minority class.
46.5
Accuracy (%)
46.0
46.0
Accuracy
45.5
45.5
45.0
44.5
44.5
0.030.040.05
(a) Hyper-parameter w
(b)
sapeer-prrametr &
46.5
46.5
46.3
46.0
46.1
Accuracy
45.5
Accuracy
45.9
45.7
45.0
45.5
45.3
44.5
45.1
0.5
1.5
0.3
0.5
1.2
1.5
1.8
2
(c) Hyper-parameter
(d) Training factor
Figure 8: Analytical experiments of different hyper-parameteed and settings proposed method on CIFAR-LT-100 with y 100: (a-c) with various hyper-parameten, (d) with the setting of training factor.

---

The hyper-parameter parameter controls the smoothness of gt: which further decides the sample weight of foreground distribution (4). As shown in Fig. [(c), the smoothness factor r I1 achieves the best performance, which indicates the inverse class frequency. Recalling that we adopt 1randam ~ Bernoulli (1) to decide whether mixing the background and foreground pair with our OTmix, where we use μl to set Vrandom ~ Bernoulli (f) and explore the effect of μ.. As plotted in Fig. g d), ours with μر 1 produces the best performance. That is to say, will randomly mix the background and foreground images with a high probability during the first half of training epochs, and we are more likely to use OTmix to mix the background and foreground pairs in the latter half of epochs.
Mixed Fractions To examine the indispensability of the images unchanged in every batch, we conduct an additional experiment with the fraction of the mixed images in every batch on CIFAR-LT- 10 and CIFAR-LT- 100 under different methods and fractions. We denote Fraction = Overall in every batch. From the results Table when the number mixed samples decreases (Fraction 4). the performance of OTmix deteriorates significantly, which suggests that it is better to use mixed samples alone in every batch our method.
Mined
Table 7: Classification errors of ResNet-32 CIFAR-LT-10 and CIFAR-LT-100 under different methods and mixed fractions.
CIFAR-LT-10
CIFAR-LT-100
Method
Fraction
100
50
100
10
ERM
26.9
22.9
12.9
63.2
56.3
43.4
ERM CMO
100%
25.0
18.6
11.5
56.1
51.7
40.5
ERM + OTmix
100%
21.7
16.6
9.8
53.6
49.3
38.4
ERM OTmix
50%
24.7
20.0
12.3
55.1
52.0
40.4
ERM + OTmix
25%
23.6
20.1
11.1
58.1
53.1
40.5
ERM OTmix
12.5%
27.2
21.8
12.1
57.1
52.7
41.1
Discussion of Confusion matrix Considering the confusion matrix played an important role in learning the cost function based class-level, we discuss the performance of our method and the computational cost of the normalized confusion matrix calculated on a balanced validation set, an imbalanced training set, and small balanced subset sampled from imbalanced training set in Table Meanwhile, the confusion matrix is represented as two states, fixed and adaptive. The former denotes where the confusion matrix remains unchanged in approach and the latter changes dynamically in each epoch. From Table we can draw a few observations: methods perform significantly better overall and few than the fixed methods. (2) Compared with the fixed balanced training setting fixed: the fixed imbalanced training setting Dijieed is preferable by providing more sample information fixed > Dbat Diree) However, large-scale samples can drastically increase the computational cost, making it difficult to implement adaptively. (3) Despite the time spent in the balanced training setting less, the balanced validation setting exhibits superior in terms of overall performance. To summarize, OTmix with the adaptively balanced validation setting enhances the suitability of OTmix for long-tailed classification.
(1) With the same settings, the adaptive
Table Classification errors of the ERM+OTmix with different confusion matrices under various methods and calculated settings on iNaturalist 2018.
Method
Setting
ALL
Many
Medium
Few
Time
Fix
Balanced validation
31.0
29.3
29.5
33.3
86s
Fix
Imbalanced training
31.5
31.0
30.2
33.2
1277s
Fix
Balanced training
31.8
32.7
30.3
33.5
58s
Adaptive Adaptive
Balanced validation (OTmix)
30.5
30.7
29.5
31.6
86s
Balanced training
31.1
32.9
30.6
31.1
58s

---

7.4 Computational Cost
The optimal transport (OT) problem our method between probability distributions is computed by the Sinkhorn algorithm [68], which introduces the entropic regularization term for fast computation. To compute the OT distance between dimensional discrete distributions, the Sinkhorn algorithm requires the computational cost of
reach e-accuracy. In our case, corresponds to the batchsize, which is set to 128 in our experiments. We compare the computational cost of different methods on a Pentium PC with a single GTX 3060 GPU.
Table 9: Computational cost (s) per training epoch on long-tailed datasets.
Method
CIFAR-LT-10 CIFAR-LT-100
ImageNet-LT iNaturalist 2018
ERM
2.85
2.17
310.47
1251.29
ERM+CMO 3.35
2.85
319.73
1360.66
ERM+OTmix 4.59
3.79
333.64
1382.54
In addition, we also report the computational cost (s) per training epoch on long-tailed and balanced datasets, respectively. As shown in Table|Pand Table mixup-based methods usually take more time than ERM. is reasonable since mixup-based methods need to mix images. Besides, OTmix spends more time than CMO since solve OT problem to pair background image and foreground image. Still, introducing OTmix to existing mixup-basee basd--aaddde-e methods in balanced classification consumes more time. However, it is worth noticing that only need to employ the OTmix during the late half phase of the training process. In summary, combining ours with others produces a better performance on long long-tailed and balanced datasets with an acceptable cost.
Table 10: Computational cost (s) per training epoch on balanced datasets.
Method
CIFAR-10
CIFAR-100
ERM
11.9
12.8
ERM+Mixup
12.1
13.8
ERM+OTmix (Mixup)
17.1
18.1
ERM+Cutmix ERM+OTmix (Cutmix)
13.2
15.7
19.1
21.5
SaliennnieccyMix
14.2
17.9
ERM+OTmix (SaliencyMix)
21.5
24.9
7.5 More Visualization Results and Analysis
Statistical Results of Mixed Images To intuitively reveal that OTmix is more effective than CMO, we show the statistical results of the mixed images generated by CMO and OTmix on CIFAR-LT-I0. respectively. Specifically, we summarize the 10 x 10 matrix m for the ten-class classification task in one training epoch, where element mi denotes the number of pairs between the foreground images from the i-th class and the background images from the j-th class. As shown in Fig. (a)) we can see that regardless of the foreground image from which class, CMO mainly mixes it with the background image from the majority classes. For example, the foreground images from the "truck" will be mixed with the background images from the "airplane" class, even the truck more similar the automobile. However, Fig. ( indicates that our proposed method builds more reasonable pairs by selecting the most relevant background image for each foreground image. For example, the "horse" more easily confused with "deer" than "airplane". These results validate that ours can provide more reasonable generated samples than CMO.
To gain more intuitive insight into the dynamic changes within the mixing process of our method, we provide more visualization results of mixed image pairs INatualist 2018 in Fig. T0 The foreground image and the selected background image commonly have significant semantic similarity. suggests that OTmix has the capacity to generate reasonably mixed samples for long-tailed classification.
Visualization Results of Mixed Image Pairs
5

---

+95
siepland
192
Foreg rrunddddd labels
515
275
387
88
37
18
12
400
277
22
187
15
SLB
102
49
469
522
182
101
21
16
321
90B
0
300
Foreground
deet
372
1233
600
103
510
362
87
41
24
14
200
0
400
1200
ship
1/3
101
Ireett
22/9
arptare
actmmnne
Background labels
Background labels
(a) CMO
(b) OTmix
Figure 9: Image mixing statistical results of OTmix and CMO CIFAR-LT-10 with o 100.
Foreground
Rona Aurora
Lycindes
Baribia Imbricata
Syncerus Cafler
Flavaconis
Amanita
Abudefdut Saxatills
Latredectus Hesperus
Nereocystis
Lasthroux
Background
Leucurus
Antllocapra Americana
Raua Drasteall
Achlyodes Pallida
Misissippiemii
Alligaator
Bison
Amanita Muscaria
Abadeffuu Sordidus
Leucauge Venusta
Postelsia Palmaeformis
Mixed
Figure 10: The visualization results of image pairs used to semantically meaningful mixed images iNaturalist 2018.
7.6 Negative Societal Impacts and Limitations
This work develops a simple and effective image-mixing method for long-tailed learning, which has the potential to encourage researchers derive new and better methods for the line of mixing images or long-tailed learning. However, if there is a sufficiently malicious or ill-informed choice of a long- classification task an mage--ixxing task, it may indirectly lead negative impact. Employing an imprecise or incorrect confusion matrix can mislead our method to build unsatisfactory pairs for image-mixing.


Reviews:
[{"review_id": "UwA87aEZhA", "reviewer": "Reviewer_ARR2", "summary": "To tackle the data imbalance problem,  the authors suggest to manipulate the data distribution by introducing a new data augmentation mechanism.  This mechanism is a guided mix-up operation that uses backgrounds mostly from major classes and foregrounds from low frequency classes. To make the augmented images look more reasonnable, the authors suggest to match images based on a similarity at sample and/or class level. They formulate the  matching problem as a transport problem. The performance of the introduced augmentation is validated by several experiments on different benchmarks as well as balanced datasets.   ", "questions": "Did the authors also consider to keep a fraction of the images unchanged in every batch? I am wondering if the trained networks will perform better in this case on real images than if they have been exclusively trained on mixed images ", "limitations": "na", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Although this paper introduces a simple extension of the mixup operation, the intuition behind it makes it an elegant solution. The formulation is technically sound and the claims are sufficiently supported with experiments. The paper is also well-written. ", "weaknesses": "minor weakness: some typos that can be fixed easily", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**(1) Response to typos:** Thanks for your reminder and we will carefully address them in the revised version.\n\n**(2) Response to the fraction about the mixed images:** According to your suggestion, we conduct an additional experiment with the fraction of the images unchanged in every batch in the pdf uploaded on the ''global'' response. We denote $Fraction = \\frac{Mixed}{Overall}$ in every batch. From the results in Table 1, when the number of mixed samples decreases ($Fraction\\downarrow $), the performance of OTmix deteriorates significantly, which suggests that it is better to use mixed samples alone in every batch."}, {"review_id": "nrgQJPJ9bh", "reviewer": "Reviewer_YYXw", "summary": "The authors tackle the notable challenge of learning from imbalanced data, proposing an adaptation to mixup-based or copy-paste-based data augmentation for rare categories that takes sample-level semantic similarity into account. Their proposed method uses optimal transport to underpin an adaptive mixup strategy, combining sample- and image-level information. They show this context-aware mixup strategy is useful as data augmentation for both long-tailed and balanced datasets.\n\nAfter reading the authors rebuttal and discussing with them, I will maintain my score of 6.", "questions": "Bias and imbalance is often as related to where data is captured as it is to what is captured. How might bias in image backgrounds towards specific parts of the world, for example, impact the effectiveness of this type of approach?", "limitations": "Little to no discussion of limitations or failure modes of this method, it would improve the quality of the work to understand where this helps and where it does not. Does performance for any rare category get worse?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The method is intuitively motivated, the implementation is reasonably simple, and the results are good and demonstrated on real-world imbalanced datasets such as iNaturalist. The authors further demonstrate improvement over vanilla mixup for balanced datasets like CIFAR.", "weaknesses": "Though they emphasize the semantic quality improvement of their method, it’s clear that this could perhaps be taken even further. For example, the method places pandas (a bear) on top of images of other bears, which is reasonable at a surface level. But these bears have drastically different habitats, at a more nuanced level it’s fundamentally unreasonable to see a panda and a black bear in the same habitat.\n\nPerformance is significantly worse on common categories in iNaturalist when using this method than ERM+CMO, and as common classes are the vast majority of what is seen in the real world, the drop in overall accuracy of this method when deployed on imbalanced test data might be difficult to overcome. \n\nNits:\n\nGrammar is often incorrect throughout, too many small errors to reasonably capture here\n\nIt would be good to reference https://arxiv.org/abs/2302.11861, which shows targeted copy-paste augmentation aligned with a novel test domain can improve out of domain robustness, a similar finding, as well as https://arxiv.org/abs/1904.05916 which shows rare-class performance improvement from synthetic data including a copy-paste baseline which includes context of night vs day backgrounds.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**(1) Response to unreasonable samples:** Thanks for your insightful comments. We agree that for any imperfect method, there can be cases where foreground patches are mixed unreasonably with background patches, like the example you pointed out. We would like to point out that CMO mixes patches completely randomly while ours does it guided by an adaptive OT transport plan. Therefore, we believe that the overall proportion of unreasonable mixed images by ours should be less than that of CMO, which explains that ours outperforms CMO. To empirically verify this, we leverage the method of data valuation[1], which assesses the quality of data to see whether a training dataset can improve model performance or mitigate undesirable biases. Specifically, we generate 512 mixed samples as training data and randomly select 128 images from the test split of CIFAR-LT-100 as test data by OTmix and CMO, respectively. We then use the data valuation algorithm in [2] implemented in pyDVL to examine the quality of the mixed images of the two methods. The comparison result of the average data values (larger is better) is OTmix=0.198 and CMO=0.115. It indicates that OTmix generates higher-quality mixed images than CMO. In the emerging area of using mixed images for long-tailed problems, we agree that there is surely room for further improvement, which will be studied in our future work.\n\n[1] OpenDataVal: a Unified Benchmark for Data Valuation. [2] Data Shapley: Equitable Valuation of Data for Machine Learning\n\n**(2) Response to the performance:** We agree that our performance on common (Many) categories is not as good as ERM+CMO's on iNaturalist. However, we have significant improvements in overall, medium, and few categories on CIFAR-LT-10, CIFAR-LT-100, ImageNet-LT, and iNaturalist 2018 datasets. These experiments demonstrate the overall effectiveness of our OTmix for imbalanced classification.\n\n**(3) Typos:** We will carefully address the grammar issues in our revision.\n\n**(4) Response to references:** Thanks for your references, which are quite inspiring. [1] proposes a targeted augmentation, Copy-Paste (Same Y), which randomizes backgrounds between cameras in similar habitats, where the relevant dataset to long-tailed datasets is species classification from camera trap images (IWILDCAM2020-WILDS). [2] proposes CCT-20 dataset come from the Caltech Camera Traps (CCT) dataset, where cis-locations are defined as cations seen during training and trans-locations as locations not seen during training. Both papers are used to solve multi-domain unbalanced classification problems, which are related but also differ from our task, the single-domain unbalanced classification problem. The dataset in [1] is quite large and [2] does not disclose the code or dataset. Therefore, it would be a bit hard for us to conduct experiments to compare with them in the rebuttal phase. We will put them into our reference and provide a more detailed study in our revision.\n\n[1] Out-of-Domain Robustness via Targeted Augmentations. [2] Synthetic Examples Improve Generalization for Rare Classes.\n\n**(5) Response to bias and imbalance:** To empirically explore the impact of bias in image backgrounds towards specific parts of the world on our method, we conduct an additional experiment of classification accuracy in the group-oblivious setting of ResNet-50 on Waterbirds[1] in Table 2 in the pdf uploaded on the ''global'' response. \n\n**Dataset.** Waterbirds is a biased dataset in image backgrounds, whose task is to distinguish whether the bird is a waterbird or a landbird. According to the background and label of an image, this dataset has four predefined subpopulations, i.e., ''landbirds on land (Group 0)'', ''landbirds on water (Group 1)'', ''waterbirds on land (Group 2)'', and ''waterbirds on water (Group 3)''. In the training set, the largest subpopulation is Group 0 with 3,498 samples, while the smallest subpopulation is Group 2 with only 56 samples. We emphasize that the training set is biased, whereas the validation set and the test set are balanced. \n\n**Evaluation metrics.** Following [2], we report the average accuracy and worst-group (group 2) accuracy, with a focus on the worst-case accuracy, which is more important than the average accuracy in some application scenarios ad stated by [2]. \n\n**Baseline Methods.** We consider several baseline methods: (1) ERM (the cross-entropy loss); (2) CMO; (3) Focal loss, which weights well-classified samples; (4) Methods of distributionally robust optimization (DRO) for group shifts: $\\chi ^{2}\\mathrm{-DRO}$[3] and $\\chi ^{2}\\mathrm{-DORO}$[4].\n\n**Results.** Table 2 demonstrates that ours achieves a desired trade-off between the average and worst-case accuracy, a well-known challenge [2]. Besides, the worst-group accuracy of ERM+OTmix outperforms other competing methods. Specifically, ERM+OTmix improves 18.3\\% about the worst-group accuracy than ERM+CMO and 3.4\\% than $\\chi ^{2}\\mathrm{-DORO}$. Although ERM+OTmix about the average accuracy is 1.1\\% lower than ERM, the performance of other methods also declined, where ours outperforms Focal loss and two methods of DRO. These results illustrate that  OTmix can still perform effectively on the biased dataset.\n\n[1] Distributionally robust neural networks.\n[2] UMIX: Improving Importance Weighting for Subpopulation Shift via Uncertainty-Aware Mixup. [3] Large-scale methods for distributionally robust optimization.\n[4] Doro: Distributional and outlier robust optimization.\n\n**(6) Response to limitations:** We have discussed negative societal impacts and limitations in Appendix 7.6. The imprecise or incorrect confusion matrix can mislead our method to build unsatisfactory pairs for image-mixing and to generate invaluable minority samples. So we need to provide a trained model or train a model from the beginning until the confusion matrix can be sufficient to provide the sample-level information for our method. It may spend some time."}, {"review_id": "sS9TMH5kJ8", "reviewer": "Reviewer_HKTJ", "summary": "This paper addresses the interesting long-tailed classification problem, which is usually confronted by real-world machine learning tasks. In particular, this work extends the existing idea of mixing the background images from the majority class and foreground images from the minority class for imbalanced classification by incorporating class-level and sample-level information via the newly developed optimal transport-based image-mixing (OTmix) method. This paper has conducted extensive experiments on artificially crafted and real-world long-tailed datasets, with results demonstrating that OTmix outperforms various types of approaches designed for long-tailed classification, including augmentation-based mechanisms, and cost-sensitive methods, among many others.", "questions": "1. In P3 L130, this paper mentions that \"which replaces the image region with a patch from another training image\". I wonder what is the patch size used in this paper. Do authors consider the effect of patch size on classification performance? To answer this question, I recommend conducting another ablation study by utilizing various batch sizes. e.g., 4x4, 8x8, or 16x16.\n2. How this study works during the inference stage? Should the data in the test set also mix the background and foreground images? If not, could you explain why image-mixing methods can improve the classification performance on the imbalanced dataset?", "limitations": "See Weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and presents information in a coherent manner, making it easy to comprehend.\n2. The introduced OTmix demonstrates the capability to enhance classification performance on imbalanced datasets. Additionally, it can serve as a valuable data augmentation technique for balanced datasets.\n3. The empirical evaluation in this study holds some validity. Extensive experiments have been conducted on diverse long-tailed datasets, and the results provide validation for the superiority of the proposed OTmix compared to state-of-the-art competitors. Furthermore, the ablation studies discussed in Section 5.2 illustrate the importance of each technical contribution made in this research.", "weaknesses": "### Major\n\n1. The technical contribution of this research is somewhat constrained. While this study expands on previous methods by incorporating sample-level and class-level information, its technical advancements largely rely on established image-mixing approaches. Hence, it can be considered more of an incremental contribution.\n2. This work would benefit from including a baseline comparison with well-known studies in long-tail classification tasks, such as Focal Loss [1] and Class-Balanced Loss [2]. This would provide a more comprehensive evaluation of the proposed approach.\n3. The rationale behind leveraging the confusion matrix to learn class-level information is insufficiently explained, making it challenging for readers to comprehend how the confusion matrix can be advantageous for long-tailed classification tasks. Further clarification is needed to elucidate the benefits and implications of utilizing the confusion matrix in this context.\n\n\n\n[1] Focal Loss for Dense Object Detection, https://arxiv.org/abs/1708.02002\n\n[2] Class-Balanced Loss Based on Effective Number of Samples, https://arxiv.org/abs/1901.05555\n\n### Minor\n\n1. In P4 L148, it is advisable to provide a concise overview of the proposed framework immediately following \"Fig 2.\" This will aid readers in quickly understanding the significant contributions of this research.\n2. Section 5.4 should also conduct explements for exploring the hyperparameter sensitivity of $w$ in Eq (7) and $\\lambda$ in Eq (9) on classification performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**(1) Response to contribution:** Please see global response.\n\n**(2) Response to the comparison with well-known baselines:** Thanks for your suggestion. We provide more comprehensive evaluation experiments with Focal loss and Class-Balanced (CB) loss on long-tailed datasets in the pdf uploaded on the ''global'' response, where Table 3 lists the results of the Focal loss and Table 4 reports the results of the Class-Balanced loss. From the results in Table 3, we observe that Focal + OTmix achieves better performance than Focal and Focal + CMO on CIFAR-LT-10 and CIFAR-LT-100 with different imbalance ratios. Table 4 shows that ours obviously outperforms other methods (CB and CB+CMO) on CIFAR-LT-10, including softmax and sigmoid. These experiments demonstrate the effectiveness of the proposed OTmix in Focal loss and CB loss. We will add these results in our revision if necessary.\n\n**(3) Response to the reason for using the confusion matrix:** Thanks for your comments. This work aims to learn the optimal transport plan between the background images and foreground images. The cost function $C_{ij}$ measures the distance between the $i$-th background image and the $j$-th foreground image, which plays an important role in learning the optimal transport plan. Generally, the cost function $C_{ij}$ should be small if the $i$-th background image and $j$-th foreground image are close, and a small cost function $C_{ij}$ can result in a large transport plan element $T_{ij}$, where the image $i$ is thus more likely to mix with the image $j$. Therefore, we need to design a proper cost function. A simple but straightforward way is defining the sample-level function using the features of $x_i$ and $x_j$. However, this way ignores the class information of $x_i$ and $x_j$. For example, it is reasonable that the class about cat is close to the class about dog than the class about airplane. Therefore, we set the diagonal element as 0 to achieve the normalized confusion matrix, whose element $\\hat{F}\\_{y_i, y_j}$ shows the class similarity between  $y_i$ and $y_j$. After that, we define the cost function as $C_{ij} =1-\\hat{F}\\_{y_i, y_j}$.  When the class of $y_i$ is most easily confused with the class of $y_j$ in the confusion matrix, $C_{ij}$ would be small and $T_{ij}$ would be large, where we are inclined to mix the image $x_i$ and image $x_j$. So designing the class-level cost function using the confusion matrix is beneficial for us to implement the image-mixing process. \n\n**(4) Response to the concise overview of the proposed framework:** Following your suggestion, we provide a concise overview of the proposed framework as below. We consider a distribution $P_\\mathrm{back}$ over the background images mainly from majority classes and another distribution $Q_\\mathrm{fore}$ over the foreground images mainly from minority classes. By minimizing the OT distance between these two distributions, where we design the sample-level and class-level cost function, we use the learned transport plan as guidance to create the mixed images between the two distributions.\n\n**(5) Response to the hyperparameter sensitivity of $\\omega $ and $\\lambda$:** Thanks for your comments. The sensitivity of the hyperparameter $\\omega $ in Eq (7) on classification performance has been discussed in Fig.6 (a) in Appendix 7.3. In terms of the combination ratio $\\lambda$ in Eq (9), it is sampled from the beta distribution $\\mathrm{Beta} (\\alpha, \\alpha)$ (see line 133 in our submitted version). We have discussed the effect of the hyperparameter $\\alpha$ in Fig.6 (b) in Appendix 7.3. Besides, we also explore the experiments about the hyperparameter $r$ and the training factor $\\mu $ in Fig.6 (c) and Fig.6 (d) in Appendix 7.3, respectively. We will make it more clear in our revision.\n\n**(6) Response to patch size:** Thanks for your comments. We kindly remind you that we do not use a fixed patch size (i.e., the size of the bounding box) during the training stage, where we randomly sample the bounding box following the Cutmix in each mini-batch.\n\nIn Appendix 7.1, we have provided the details of the bounding box, which is calculated in Eq (11). Specifically, the bounding box coordinate is $\\mathbf{B}=\\left(r_x, r_y, r_w, r_h\\right)$. Here, $r_{x}$ and $r_{y}$ are randomly sampled from the uniform distribution, $r_{w}$ and $r_{h}$ depend on the combination ration $\\lambda $, where $\\lambda $ is sampled from the beta distribution $\\mathrm{Beta} (\\alpha, \\alpha)$. Therefore, different bounding boxes can be randomly generated to improve sample diversity in each mini-batch. Besides, we have performed the ablation study about the hyperparameter $\\alpha$ in Fig.6 (b) in Appendix 7.3.\n\n**(7) Response to the test image:** Thanks for your comments. During the inference stage, we do not need to mix the background and foreground images. Following the standard classification pipeline, the sample in the test set will be directly fed into the feature extractor followed by the classifier to predict the label of the input. That is to say, we only mix the images in the training stage based on OTmix. It provides an adaptive way to learn the similarity between background and foreground images. Now, the rich context of the majority classes as background images can be better transferred to the minority classes as foreground images. Since we augment more semantically reasonable and meaningful mixed images to train the classifier, OTmix improves the classification performance on the imbalanced dataset. Extensive experiments also demonstrate the effectiveness of our method for long-tailed classification."}, {"review_id": "fweA7Fgvu9", "reviewer": "Reviewer_mXEg", "summary": "This paper aim to solve long-tailed problem in the perspective of data augmentation. Specifically, the backgrounds of majority classes are mix with the foreground of minority classes. This work found out that randomly mixup images might face semantic misalignment, which impedes the effectiveness of this scheme. Therefore, this work use optimal transport to pair images with similar semantics. The paired images are mixed to form new training samples. On several benchmarks, this approach obtain good performance.", "questions": "The complexity of estimate $T$", "limitations": " ", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper has following strengths:\n\n- This paper identity a issue of former mixup-based long-tailed method that the mixed samples are semantically mismatched. This paper proposes a sound solution to tackle this issue.\n\n- This paper is easy to follow.\n\n- This method can be unifed with former LT methods as a plug-in module.\n\n- This paper has a clear motivation.", "weaknesses": "This paper has following weaknesses:\n \n- The performance gain on RIDE (3 experts) compared with CMO is limited.\n\n- The contribution is limited since it's a updated version of CMO.\n\n- The key point is to find semantical similar pairs, it's crucial to visualize some pair results.\n\n- OT is to calculate optimal plan for aligning distributions. But the ideal solution for a minority sample is to find its cloest minority neighbor. Does using OT also lead to a suboptimal solution?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**(1) Response to performance gain:** We would like to emphasize that in the highly competitive area of imbalanced classification, our performance gain over CMO is consistent and significant. Specifically, on CIFAR-LT-10 and CIFAR-LT-100 (imbalance factor=100,50,10), and ImageNet-LT (imbalance factor=256), the performance improvements of our method+RIDE over CMO+RIDE are encouraging, i.e., 0.5\\%, 0.6\\%, 0.8\\% on CIFAR-LT-10, 0.7\\%, 0.8\\%, 0.7\\% on CIFAR-LT-100 and 1.1\\% on ImageNet-LT. Even on iNaturalist 2018, the performance gain of our method+RIDE over CMO+RIDE is 0.2\\%, which is the smallest but still consistent. Besides, ours+ RIDE can achieve better performance in the case of few classes than CMO+RIDE, such as the surprising performance gain of 8.5\\%. Moreover, as a data augmentation method, ours can be naturally combined with many existing long-tailed methods or architectures. As summarized in Tables 5 and 6 of Appendix 7.2, we compare ours with CMO when combined with other loss functions, where ours significantly outperforms CMO with various losses. Our OTmix can also be used for balanced classification as a general data augmentation method, while CMO becomes the standard CutMix in the balanced classification task. Ours can be combined with classical mixup-based methods to achieve better performance in balanced classification.  \n\n**(2) Response to contribution:** Please see global response.\n\n**(3) Response to visualization:** Following your suggestions, we have provided the visualization of image pair results on ImageNet-LT. As shown in Fig.1 of the pdf uploaded on the ''global'' response, we plot the more semantically reasonable and relevant background image for each foreground image given the learned transport plan. For example, the semantic information of ''Grey fox'' is semantically relevant to ''Dhole''. It visually reveals that our proposed method can create meaningful and advantageous samples, benefiting the balanced and imbalanced classification tasks. We will provide more visualizations in our revision.\n\n**(4) Response to suboptimal solution brought by OT:** We kindly remind you that our proposed method is not finding the closest minority neighbor for a minority sample. Instead, we aim to find the most relevant background image (mainly from majority classes) for each foreground image (mainly from minority classes). Below, we will explain it in detail.\n\nOur key concept is to paste an image mainly from a minority class (foreground) onto rich-context images mainly from a majority class (background). Even though randomly pairing the background and foreground images can generate an acceptable performance in CMO, we find that it might generate more meaningful mixed images if we consider designing the adaptive strategy to pair the foreground and background images. To this end, we propose a more adaptive image-mixing method based on optimal transport. We consider an empirical distribution $P_{\\mathrm{back} }$ over the background images and an empirical distribution $Q_\\mathrm{fore}$ over the foreground images, where the former are more likely from the majority and the latter from minority classes. To learn the semantic similarity between background and foreground images, we formulate the learning of the similarity as the OT problem between $P_\\mathrm{back}$ and $Q_\\mathrm{fore}$. In this way, we can view the learned transport plan as the similarity between foreground and background samples, which can be naturally used as guidance to select the most relevant background image for each foreground image. \n\nBesides, similar to your proposed opinion, the classical synthetic minority over-sampling technique (SMOTE) [1] uses the interpolation between a given minority sample and its nearest minority neighbors to create new samples. However, the performance of classical SMOTE is far worse than our baselines, such as Tables (1-2) in M2m [2], where M2m is inferior to ours. It thus indicates that generating the samples between the given minority sample and its nearest minority neighbors is not the ideal solution for long-tailed classification.\n\n[1] SMOTE: Synthetic Minority Over-sampling Technique. [2] M2m: Imbalanced Classification via Major-to-minor Translation.\n\n**(5) Response to the complexity of estimate T:** Thanks for your comments. The optimal transport problem in our paper is computed by the Sinkhorn algorithm [1], which introduces the entropic regularization term for fast computation. To compute the OT distance between $n$ dimensional discrete distributions, the Sinkhorn algorithm requires the computational cost of $\\mathcal{O}(n^{2} log(n) /\\varepsilon ^{2} )$ reach $\\varepsilon$-accuracy. In our case, $n$ corresponds to the batch size, which is usually set to 128 in our experiments.\n\nWe have also compared the computational cost of different methods on a Pentium PC with a single GTX 3060 GPU. We report the computational cost (s) per training epoch on long-tailed (Table 7 in Appendix) and balanced datasets (Table 8 in Appendix), respectively. \n\n[1] Sinkhorn distances: Lightspeed computation of optimal transport. "}]

Author rebuttal:
We thank all reviewers for their constructive comments. We upload a global pdf about the figures and tables. Below we first address common concerns and then respond to each reviewer.

**Response to Reviewers HKTJ and mXEg about contribution and novelty:** Learning relatively balanced classes from the data perspective is an effective solution to the long-tailed problem, where the study on mixup-based methods has gradually attracted the attention of the researchers. Among them, CMO recently constructs mixed samples between the background images more likely from the majority classes, and foreground images more likely from the minority classes. Despite the effectiveness of CMO, it may generate unreasonable mixed samples for arbitrarily pairing the foreground and background images and ignoring their semantic distances. Therefore, designing an effective method to pair the background images and foreground images is quite necessary for creating reasonable augmented samples.

To this end, we propose a more adaptive image-mixing method for generating semantically reasonable, and meaningful mixed samples based on optimal transport (OT) on long-tailed classification. It is non-trivial since we formulate the image-mixing problem as the problem of learning the optimal transport plan between two discrete empirical distributions, where one distribution is over the background images and another is over the foreground images. Due to the importance of the cost function in learning the transport plan for OT, we further design the cost function using the sample-level representation and class-level label information. Once we obtain the transport plan, we use it as the similarity guidance to select the most relevant background image for each foreground image. Besides, our proposed method can also be adapted to the balanced classification setting, where CMO becomes the standard Cutmix. It shows that our proposed method can serve as a general data augmentation method, which is effective in both imbalanced classification and balanced classification.

To summarize, CMO is the state-of-the-art method in the line of mixup-based methods for long-tailed classification. We propose a new method in this line that improves over CMO via nontrivial efforts, which we believe has significant contributions.

Meta-review: This paper proposes an adaptive image-mixing augmentation method to generate semantically meaningful images of minority classes for long-tailed classification. The proposed method employs optimal transport (OT) to incorporate class-level and sample-level information for adaptive mixing. Initially, reviewers raised concerns such as the generated image may be unreasonable, performance drops on common categories, the contribution is incremental, the performance gain is limited, lack of comparisons with some baselines, and the rationale of using the confusion matrix and OT. Although not all concerns were well addressed in the rebuttal, all reviewers recommended positive final ratings. Considering that the method is intuitively motivated, technically sound, and effective, as appreciated by reviewers, the AC follows this unanimous recommendation. Reviewers did raise valuable concerns that should be addressed. The authors are encouraged to make the necessary changes in the camera-ready version.
Decision: Accept (poster)