Title: Enhancing Minority Classes by Mixing: An Adaptative Optimal Transport Approach for Long-tailed Classification
Venue: NeurIPS 2023
Accepted: yes
Keywords: Long-tailed Classification

Optimal Transport

Image-mixing

Semantic Similarity
Abstract: Real-world data usually confronts severe class-imbalance problems, where several majority classes have a significantly larger presence in the training set than minority classes. One effective solution is using mixup-based methods to generate synthetic samples to enhance the presence of minority classes. Previous approaches mix the background images from the majority classes and foreground images from the
minority classes in a random manner, which ignores the sample-level semantic similarity, possibly resulting in less reasonable or less useful images. In this work, we propose an adaptive image-mixing method based on optimal transport (OT) to incorporate both class-level and sample-level information, which is able to generate semantically reasonable and meaningful mixed images for minority classes. Due to
its flexibility, our method can be combined with existing long-tailed classification methods to enhance their performance and it can also serve as a general data augmentation method for balanced datasets. Extensive experiments indicate that our method achieves effective performance for long-tailed classification tasks. The code is available at https://github.com/JintongGao/Enhancing-Minority-Classes-by-Mixing.

Enhancing Minority Classes by Mixing: An Adaptative Optimal Transport Approach for Long-tailed Classification

Abstract

Real-world data usually confronts severe class-imbalance problems, where several majority classes have a significantly larger presence in the training set than minority classes. One effective solution is using mixup-based methods to generate synthetic samples to enhance the presence of minority classes. Previous approaches mix the background images from the majority classes and foreground images from the minority classes in a random manner, which ignores the sample-level semantic similarity, possibly resulting in less reasonable or less useful images. In this work, we propose an adaptive image-mixing method based on optimal transport (OT) to incorporate both class-level and sample-level information, which is able to generate semantically reasonable and meaningful mixed images for minority classes. Due to its flexibility, our method can be combined with existing long-tailed classification methods to enhance their performance and it can also serve as a general data augmentation method for balanced datasets. Extensive experiments indicate that our method achieves effective performance for long-tailed classification tasks. The code is available at https://github.com/JintongGao/Enhancing-Minority-Classesby-Mixing.

1 Introduction

Large-scale balanced datasets play a vital role in the remarkable success of deep neural networks for various tasks. However, datasets in many real-world applications often exhibit unexpected long-tailed distributions where most of the data belongs to several majority classes while the rest spreads across lots of minority classes. The model trained on such a long-tailed dataset will be biased toward majority classes, leading to poor generalizations about minority classes.

Re-weighting [1, 2, 3, 4, 5, 6, 7], over-sampling [8, 9, 10, 11, 12], under-sampling [13, 14, 15, 16], data augmentation [17, 18, 19, 20, 21, 22, 23], two-stage methods [24, 25, 26, 27], and other methods [28, 29] are common solutions to the long-tailed problem. Among them, over-sampling and data augmentation aim to balance the data distribution by oversampling or generating closely related minority classes. As the representative data augmentation techniques, mixup and its variants [30, 31, 32] have performed satisfactorily in the computer vision fields, whose key idea is constructing mixed samples by performing linear interpolations between data/features and corresponding labels. Considering mixup-based methods are designed for balanced data, applying them to long-tailed classification without any adjustments may ignore the specificity of long-tailed data distribution. Recently, some novel mixup-based methods for the long-tailed problem have been explored [22, 33,

Foreground

MixedSamples

Foreground

MixedSamples

Panda

Seashore

Butterfly

CMO

RENA

Taxicab

GreyWhale

Dromedary

Eagle

Panda

Tiger

Dugong

Orca

s.no

susArctos.

BlackBear

GreyWhale

Sea Lion

34, 21]. As a representative example, Context-rich Minority Oversampling (CMO) [20] generates minority-centric images by mixing the background and foreground images, where background images are mainly from the majority classes while foregrounds are mainly from minority classes. The critical concept of CMO is to paste an image from a minority class (foreground) onto rich-context images from a majority class (background). Despite the effectiveness of CMO, it cannot be guaranteed that all the generated images are realistic and reasonable by randomly mixing the background and foreground images. For example, we visualize mixed samples created by ours and CMO in Fig. 1, where we take 'Panda' and 'Grey Whale' as two foreground images. CMO generates unreasonable mixed samples since pandas are hardly seen in the sky, the beaches, or the streets, and grey whales rarely appear in the desert, the flowers, or the sky. Therefore, instead of arbitrarily pairing the foreground and background images, it is essential to consider the semantic distances between the foreground and background images to generate more meaningful and advantageous samples.

In this work, we propose a more adaptive image-mixing method based on optimal transport (OT), which is a powerful tool for computing the distance between two distributions under the form of multiple sampling [35]. We consider an empirical distribution P back over the background images and an empirical distribution Q fore over the foreground images, where the former are more likely from the majority and the latter from minority classes. To learn the semantic similarity between background and foreground images, we formulate the learning of the similarity as the OT problem between P back and Q fore. In this way, we can view the learned transport plan as the similarity between foreground and background samples, which can be naturally used as guidance to select the most relevant background image for each foreground image. Due to the importance of cost function in learning the transport plan for OT, we further design the cost function using the sample-level representation and class-level label information. Hence, the rich context of the majority classes as background images can be better transferred to the minority classes as foreground images for providing an adaptive way to learn the similarity between them and generate more semantically meaningful mixed images. Interestingly, our proposed OTmix can not only be combined with existing long-tailed methods but also be used as a new image-mixing strategy for balanced datasets.

Our main contributions are summarized as follows: (1) We propose a novel image-mixing method for generating reasonably mixed samples for long-tailed classification, where we introduce a distribution over the background images mainly from majority classes and another distribution over the foreground images mainly from minority classes. (2) By minimizing the OT distance between these two distributions, we view the learned transport plan as similar in guiding the image-mixing process. (3) We design the cost function based on the feature information and confusion matrix to learn the desired transport plan. Extensive experiments demonstrate the effectiveness of our method for long-tailed classification and also balanced classification.

2 Related Work

Data Processing Learning relatively balanced classes from the data perspective is an effective solution to the long-tailed problem [36], which can be roughly divided into over-sampling, undersampling, and data augmentation. Since our proposed method is related to the over-sampling and data augmentation methods, we describe them in detail below. Over-sampling aims to emphasize the minority classes and increase the instance number of the minority classes [20, 8, 9]. For example, the classical Synthetic Minority Over-sampling Technique (SMOTE) [12] uses the interpolation between a given minority sample and its nearest minority neighbors to create new samples. Data augmentation is another way of data processing to compensate the minority classes by generating and synthesizing new samples [18, 23]. For example, Major-to-minor Translation (M2m) [17] defines an optimization phase to augment minority classes via translating samples from majority classes. The Meta Semantic Augmentation (MetaSAug) approach [24] is proposed to perform effective

semantic data augmentation by learning more meaningful class-wise covariance. Sample-Adaptive Feature Augmentation (SAFA) [19] aims to extract diverse and transferable semantic directions from majority classes, and adaptively translate minority-class features along extracted semantic directions for augmentation.

In addition, mixup-based methods [32, 37, 38, 30, 39, 31, 40, 41, 42], the effective data augmentation methods in balanced datasets, have been developed to solve the long-tailed problem recently and are closely related to our work. For example, Mixup Shifted Label-aware Smoothing Model (MiSLAS) [27] uses mixup in its Stage-1 training without any adjustments. Rebalanced Mixup (Remix) [21] provides a disproportionately higher weight to the minority class when mixing two samples. Uniform Mixup (UniMix) [22] adopts a minority-favored mixing factor to encourage more majority-minority pairs occurrence and is further combined with the Bayes Bias (Bayias) caused by the inconsistency of prior. Targeted copy-paste augmentation [43] aligned with the novel domain enhances out-ofdomain robustness on long-tailed camera trap dataset. [44] improves minority class performance from synthetic data by contrasting night and day backgrounds. To mitigate the overfitting issue in subpopulation shift, Uncertainty-aware Mixup (UMIX) [33] adopts the sampling uncertainty to reweight the mixed samples. CMO [20] constructs mixed samples between the background images more likely from the majority classes and foreground images more likely from the minority classes. However, it ignores the semantic similarity between images by randomly mixing them. Different from CMO, we formulate the image-mixing problem as the OT problem between two distributions and use the learned transport plan as guidance to mix images, providing a general and adaptive image-mixing method for long-tailed learning.

Loss Function Engineering Designing effective training objectives is another solution to fight against class imbalance [45, 7, 5]. Label-Distribution-Aware Margin Loss (LDAM) [1] is proposed from the view of the generalization error bound, and LDAM-DRW adopts a deferred class-level re-weighting method. Balanced Softmax (BALMS) [2] accommodates the label distribution shift between training and testing and proposes a Meta Sampler that learns to re-sample training set by metalearning. Label Distribution Disentangling (LADE) [4] loss disentangles the source label distribution from the model prediction based on the optimal bound of the Donsker-Varadhan representation. Besides, Focal Loss [46] determines the weights for samples with the sample difficulty in the object detection task, and Influence-balanced Loss (IB) [3] assigns different weights to samples according to their influence on a decision boundary. Our method can be naturally combined with many loss functions.

Other Methods Recently, two-stage algorithms have been proposed [1, 5, 24], such as MiSLAS [27]. Meanwhile, a Bilateral Branch Network (BBN) [25] unifies the representation and classifier learning stages to form a cumulative learning strategy. RoutIng Diverse Experts (RIDE) [28] uses multiple experts to reduce the variance and bias of the long-tailed classifier. In addition, some approaches employ meta-learning, such as MetaSAug [24] and BALMS [2]. Causal Norm (CN) [29] pinpoints the causal effect of momentum and extracts the unbiased direct impact of each instance.

Optimal Transport As a powerful tool, OT has been applied to generative models [47, 48, 49, 50, 51], computer vision [52, 53, 7, 54, 55, 56, 57], text analysis [58, 59, 60, 61], and etc. To the best of our knowledge, there are still very limited works for the imbalanced classification by means of OT. For example, [7] proposes a re-weighting method based on OT, and [62] introduces Optimal Transport via Linear Mapping (OTLM) to perform the post-hoc correction. Different from them, ours falls into the data augmentation group by mixing images. Another recent work is Optimal Transport for OverSampling (OTOS) [63], which moves random points from a prior uniform distribution into that of minority class samples based on OT. However, our proposed OTmix introduces OT to guide the mix between majority and minority images, which leverages the rich context of the majority classes guided by OT. Alignmixup [38] adopts OT to align two images and interpolate between two sets of features in a standard classification task, which is distinct from ours in terms of task and technical detail.

3 Background

Long-tailed Classification Given a training set D = { ( x i , y i ) } N i =1 for a multi-class problem with K classes, if each class k contains n k samples, we have that ∑ K k =1 n k = N . Without loss of generality, we can always assume n 1 ≥ n 2 ≥ ·· · ≥ n K for the long-tailed problem. Denote

Probability of Instances

CostFunction

Image-mixingProcess

Pback

Bfore

TransportPlan

Majority

Minority

Classes

Sample-level

Bback

(Head)

(Tail)

Probability of Instances

T

Qfore

Mixedsamples

Class-level

Majority

Minority

Classes

(Head)

(Tail)

the model parameterized with θ as g ( θ ) , which is trained on D with the well-known Empirical Risk Minimization (ERM) algorithm [32] ignores such class imbalance and performs poorly on the minority classes [33].

CutMix Here, we use the CutMix [37] to mix samples because of its simplicity and effectiveness, which replaces the image region with a patch from another training image. Denote x ∈ R W × H × C and y denote a training image and its label, respectively. CutMix combines two training samples ( x i , y i ) and ( x j , y j ) and generates the new sample (˜ x ij , ˜ y ij ) as follows:

where the combination ratio λ is sampled from the beta distribution Beta( α, α ) , ⊙ is element-wise multiplication, and the binary mask M ∈ { 0 , 1 } W × H indicates where to drop out and fill in from two images, 1 is a binary mask filled with ones. We provide more details about M in Appendix 7.1.

Optimal Transport OT measures the minimal cost to transport between two probability distributions [35, 64, 65, 66, 67]. We only provide a brief introduction to OT for discrete distributions and refer the readers to [35] for more details. Denote two discrete probability distributions p = ∑ n i =1 a i δ x i and q = ∑ m j =1 b j δ y j , where both a and b are discrete probability vectors summing to 1, x i and y j are the supports of the two distributions respectively, and δ is a Dirac function. Then the OT distance is formulated as follows: OT ( p, q ) = min T ∈ Π( p,q ) ⟨ T , C ⟩ , where C ∈ R n × m ≥ 0 is the cost matrix with element C ij = C ( x i , y j ) which reflects the cost between x i and y j and the transport probability matrix T ∈ R n × m ≥ 0 is subject to Π( p, q ) : = { T | ∑ n i =1 T ij = b j , ∑ m j =1 T ij = a i } . The optimization problem above is often adapted to include a popular entropic regularization term H = -∑ ij T ij ln T ij for reducing the computational cost, denoted as Sinkhorn algorithm [68].

4 Method

In this work, we propose a novel adaptive image-mixing data augmentation method based on OT for long-tailed classification, where the overall framework is shown in Fig. 2. We consider a distribution P back over the background images mainly from majority classes and another distribution Q fore over the foreground images mainly from minority classes. By minimizing the OT distance between these two distributions, we design the sample-level and class-level cost functions and use the learned transport plan as guidance for the image-mixing process to generate more semantically meaningful mixed samples between the two distributions.

Main Objective Since majority classes usually have sufficient data and rich information, it is natural to leverage the majority classes to enhance minority classes. In this work, we aim to generate minority-centric images with majority contexts, combining the background and foreground images based on CutMix. The critical question is how to pair a background image and a foreground image. The first factor of the pairing strategy is that we would expect a background image to be more likely from the majority classes and a foreground image to be more likely from the minority classes, which has been modeled in CMO [20]. However, in addition to the first one, the second factor is the semantic similarity between the background and foreground images, which has not been studied in the literature and is the main focus of our paper. Specifically, suppose a set of candidate images

already satisfy the first factor, i.e., background and foreground images are from majority and minority classes, respectively. Pairing these images arbitrarily may not guarantee that the combined images are semantically meaningful and helpful, e.g., in Fig. 1.

In this paper, we aim to generate semantically meaningful images by proposing a pairing process that satisfies the first and second factors at the same time. Our proposal is derived from a distribution-matching perspective. First, we consider the distribution P back of the background images as the empirical distribution of all the images in the training dataset, which is a (discrete) uniform distribution:

Although P back is uniformly distributed, because the majority classes have significantly more images than the minority classes, it is natural that the samples from P back are more likely to be images from the majority classes, which is what we expect. Secondly, we introduce the distribution over the foreground images, which is a discrete distribution over all the training images:

where w j is the weight of the j -th sample. In this case, we expect that a sample from Q fore is more likely to be an image of the minority classes. Thus, Q fore cannot be a uniform distribution. Accordingly, we can specify w j in several ways, such as the (smoothed) inverse class frequency [69, 70, 71], or the adequate number of samples [72]. Taking the inverse class frequency as the example, we can sample the k -th class with the following probability: q k = 1 /n r k ∑ K k =1 1 /n r k , where r > 0 controls the smoothness of q k , r = 1 indicates the inverse class frequency and r = 1 / 2 means the smoothed version, i.e., if r increases, the weight of the minority class becomes increasingly larger than that of the majority class. Finally, we can use w j = q k 1 n k if y j = k .

For now, with our construction of P back and Q fore, the first factor is satisfied. For the second factor that requires the semantic similarity between a pair of background and foreground images, we formulate it as an optimization problem of the entropic OT:

where ϵ > 0 is the hyper-parameter for the entropic constraint, C ij is the transport cost function, and the transport probability T ij satisfies Π( P back , Q fore ) := { T ij | ∑ N i =1 T ij = w j , ∑ N j =1 T ij = 1 N } . Notably, as an upper-bounded positive metric, T ij indicates the transport probability between the i -th background image and the j -th foreground image, which can be naturally used to measure the importance of each background image for the foreground image when performing image-mixing.

Cost Function C ij measures the distance between ( x i , y i ) in P back and ( x j , y j ) in Q fore. As the main parameter for defining the transport distance between probability distributions, C ij plays an important role in learning the optimal transport plan, which can be flexibly defined in different ways. For clarity, we can reformulate the concerned model g ( θ ) as g 2 ( g 1 ( θ 1 ); θ 2 ) , where g 1 ( θ 1 ) denotes the feature extractor parameterized with θ 1 and g 2 ( θ 2 ) is the classifier parameterized with θ 2 . We explore a few conceptually intuitive options of C ij . A simple but straightforward way is defining the sample-level C ij with the features of x i and x j :

where cos ( · , · ) is the cosine similarity, z i = g 1 ( x i ; θ 1 ) ∈ R e and z j = g 1 ( x j ; θ 1 ) ∈ R e denote the e -dimensional representation of x i and x j , respectively. Now C ij will be small if two samples have similar or close features, where the cost is influenced by the feature extractor θ 1 . Besides, we can also define the class-level C ij only using the label information. Specifically, a normalized confusion matrix F ∈ R K ∗ K ≥ 0 can be estimated with an unbiased validation set or a small balanced subset sampled from D . Summarizing the prediction results of the target model g , the confusion matrix F shows how the model g is confused when it makes predictions. Since we focus on combining the foreground images mainly from minority classes and background images mainly from minority classes, we can set the diagonal element in F to 0 , denoted as ˆ F , to avoid mixing the images from the same class. After that, we can define C ij with the ground-truth labels of two samples:

where ˆ F y i ,y j indicates the element of row y i , column y j , of the matrix ˆ F . If y i and y j are from the same class, ˆ F y i ,y j = 0 and C ij = 1 ; if y i and y j are not from the same class but are easy get to confused by the model g , ˆ F y i ,y j would be large and C ij would be small. Now, C ij is affected by the feature extractor θ 1 and classifier θ 2 . Intuitively, based on the features and labels of samples, the cost function can also be defined as:

̸

where ω is a hyper-parameter for balancing the feature and label information and C ij will be small if two samples have similar features and are from two confused classes but y i = y j . Once the cost function C ij is defined, it can be fed into (4) for learning the transport plan T ij between foreground images and background images, where a small C ij tends to produce a large T ij .

Image-mixing Process By minimizing the OT problem, the resultant transport plan T provides an adaptive way to weigh the similarity between background and foreground images. Therefore, we can select the most suitable background image x j ′ for the foreground image x i as follows:

We aim to generate new mixed samples by combining the foreground image x i and the most relevant background image x j ′ and their corresponding labels. As their name implies, x j ′ is used as a background image while x i provides the foreground patch, which can be pasted onto x j ′ . Recalling CutMix in (1), the detailed mixing process is expressed as:

Training Details During the training process, we adopt a mini-batch setting to integrate our proposed OTmix with deep neural networks, where we learn T and θ alternatively. More specifically, at each iteration, we can sample a mini-batch B fore from Q fore and another mini-batch B back from P back to build ˆ Q fore and ˆ P back, both of which have M samples and M < N . In step (1), we can minimize the OT distance between ˆ Q fore and ˆ P back with (4) to learn the transport plan T . In step (2), we can use the learned transport plan to select the most relevant background image x ′ j for each foreground image x i based on (8). In step (3), we can generate the mixed samples with (9) and minimize the classification loss to train the model g parameterized by θ :

Since the cost function relies on either the parameter θ 1 in the feature extractor or the parameter θ in the whole model g , it might be inaccurate in the early training stage, resulting in an undesired transport plan matrix. Consequently, to avoid selecting the unsatisfactory background image for the foreground image and generating harmful mixed samples, it is more beneficial to randomly select x ′ j for x i in the early stage. To this end, we adopt y random ∼ Bernoulli ( t T ) to decide whether randomly mixing the background and foreground pair, where T is the number of training epochs and t indicates the current epoch. In the early stage, such as t ≤ T/ 2 , we can get y random =0 with a high probability, where we randomly mix the background and foreground images like CMO [20]; in the late stage, such as t > T/ 2 , we are more likely to sample y random =1 , where we can learn T with (4) and select the most relevant background image for each foreground image. Moving beyond CMO, we introduce a more adaptive and elegant image-mixing data augmentation method for long-tailed classification. We summarize our proposed method in Algorithm 1 and highlight steps (1), (2), and (3).

Using OTMix for Balanced Classifications In this case, our method can be viewed as a better alternative to other augmentation methods that mix images randomly [37, 30]. Applying OTMix in balanced classifications is similar to doing it in the imbalanced problems except that we construct both P back and Q fore as uniform distributions.

5 Experiments

5.1 Experimental Settings and Implementation Details

Datasets Following [23, 25, 27, 20], we evaluate our method on long-tailed classification benchmark datasets: CIFAR-LT-10 [73], CIFAR-LT-100 [73], ImageNet-LT [74], and iNaturalist 2018 [75]. For

clarity, the imbalance factor is defined as the data point amount ratio between the most frequent and the least frequent classes, i.e., φ = n K n 1 . Among them, CIFAR-LT-10 (CIFAR-LT-100) are created from CIFAR-10 (CIFAR-100) [76] with φ = { 100 , 50 , 10 } , respectively. ImageNet-LT, containing 1,000 classes with φ =256 , is created from ImageNet [74]. Different from them, iNaturalist 2018 is a real-world and large-scale imbalanced dataset with 8,142 classes and φ =500 .

Baseline Methods We consider several baseline methods: (1) Empirical risk minimization (ERM), training on the cross-entropy loss; (2) Mixup-based augmentation methods: Remix [21], Unimix [22], CMO [20]; (3) Other augmentation methods: M2m [17], Open-sampling [18], FSA [23], SAFA [19]; (4) Losses designed for long-tailed classifications: LDAM[1], DRW [1], BALMS [2], IB [3], LADE [4]; (6) Other methods: BBN [25], RIDE [28], Decouple [26], MisLAS [27].

Implementation Details For all datasets, we use PyTorch [77] and SGD optimizer with momentum 0.9. Besides, we set ω for defining cost function in (7) as 0 . 05 , ϵ for entropic constraint in (4) as 0 . 01 , r for the smoothness in (3) as 1 , and α for sampling combination ratio as 4 . For CIFAR-LT-10 and CIFAR-LT-100, we use ResNet-32 [78] as the backbone following [1] and use 240 epochs on a single GTX 2080Ti and set the initial learning rate as 0.1, which is decayed by 0.1, 0.1, and 0.01 at the 100th, 160th, and 200th epochs. For the ImageNet-LT, we employ ResNet50 as the backbone following [20] and use 200 epochs on four GTX 2080Ti GPUs. The learning rate is initialized as 0.1 and decays by 0.1, 0.1, and 0.01 at the 40th, 80th, and 160th epochs. For the iNaturalist 2018, we use ResNet-50 as the backbone fol-

lowing [1] and train 210 epochs on four Tesla A100 GPUs with an initial learning rate of 0.1, which is decayed by 0.1 in the 30th, 80th, 130th, and 180th epochs. Performances are mainly reported as the overall top-1 errors (%). Following [79], we also report the error rates on three disjoint subsets: many-shot classes with more than 100 training samples, medium-shot classes with 20 to 100 samples, and few-shot classes with 20 samples.

5.2 Experiments on Long-tailed Classification

We conduct experiments on CIFAR-LT-10 and CIFAR-LT-100 of ResNet-32 in Table 1, where OTmix+ERM achieves better performance than the mixup-based augmentation methods with ERM loss, especially for CMO+ERM. It confirms the validity of selecting a suitable pair to mix. Besides, OTmix with ERM loss produces comparable performance to complex long-tailed classification methods, such as specially designed losses and other augmentation methods. Considering OTmix can also be combined with some existing long-tailed losses and model architectures, we further combine it with LDAM-DRW loss [1], BALMS loss [2], and also the network with multiple experts (RIDE) [28]. It is clear that OTmix equipped with BALMS loss, DRW loss, and RIDE outperforms most of the competing methods except on CIFAR-LT-10 with φ = 10 , where it is only weaker than MisLAS. These results illustrate the superiority of the proposed OTmix.

Wefurther perform experiments on large-scale imbalanced ImageNet-LT and iNaturalist 2018 datasets. Table 2 demonstrates that OTmix based on ERM loss can still produce a comparable performance when compared with competing long-tailed losses, related mixup-based, and other data augmentation methods. Similarly, combining ours with other losses or model architecture can overall outperform these related baselines on ImageNet-LT and iNaturalist 2018, where RIDE (3 experts) + OTmix performs best. Furthermore, our method can improve the performance of the medium and few classes, which is the goal of long-tailed methods. We compare OTmix with mixup-based augmentation methods under various loss functions in Appendix 7.2 and computational cost in Appendix 7.4.

5.3 Experiments on Balanced Classification

To validate whether our proposed OTmix can be used for the balanced classification task, we adopt the commonly-used CIFAR-10 [76] and CIFAR-100 [76]. Notably, our proposed OTmix can be flexibly combined with the classical mixup-based methods when mixing images, where we consider Mixup [32], Cutmix [37], and SaliencyMix [30]. We train the networks using the same details as on the CIFAR-LT. At each training iteration, we randomly sample two mini-batches, i.e.,

B fore and B back, from the balanced training dataset. Table 3 lists the experimental results on balanced CIFAR datasets, where OTmix produces better performance. It indicates the benefit of building a better image pair to perform image-mixing data augmentation for the balanced classification task. Moreover, OTmix (SaliencyMix) achieves the best top-1 error of 5.1% and 21.7% on CIFAR-10 and CIFAR-100 datasets, respectively. These results show that OTmix is still effective on balanced

classification tasks and can be combined with some advanced mixup-based methods, proving its flexibility and availability.

5.4 Analytical Experiments

Here we implement the ablation study under various settings on CIFAR-LT-100 with φ =100 and summarize the results in Table 4. Our best-performing method uses the combined cost function with cosine similarity, sets the diagonal element in the confusion matrix F as 0, and relies on the maximum value in the transport plan matrix to select the background image for each foreground image. To reveal the influence of the cost function, we denote the cost function by only using the sample-level cost and the class-level cost as 'Sample-level cost only' and 'Class-level cost only', respectively. Besides, we also consider using euclidean distance rather than cosine similarity to define the cost function called 'Euclidean distance'. Compared with ours, which combines the sample-level and class-level costs and uses cosine similarity, all these variants perform slightly worse, demonstrating the benefit of combining features and labels to compute the cost function and the effectiveness of cosine similarity.

To avoid mixing the background and foreground images from the same category, we set the diagonal element in F as 0 and use the resultant ˆ F to compute the cost function. Here, we consider its variant by calculating the cost function using the original confusion matrix F , denoted as 'Keeping same category', which can be found to outperform by OTmix. It shows that avoiding mixing samples from the same class is beneficial, no matter from minority or majority classes. Besides, we

adopt the multinomial distribution, denoted as 'Probability selection', to select the corresponding background image for the i -th foreground image, whose probability vector can be formulated as p = softmax ( T i, 1: M ) . We observe that selecting background images according to probability distribution is inferior to using the maximum value, where the latter chooses the most relevant background image for each foreground image mainly from minority classes without randomness. It proves the necessity of using the maximum value in the transport plan to build a pair for mixing.

̸

Furthermore, we use the confusion matrix itself ('Confusion matrix' in Table 4) to design the imagemixing pair instead of OT. The confusion matrix can be from a pre-trained model on the imbalanced training set with standard ERM loss. Specifically, after sampling the background and foreground images, if the class k 1 of the foreground image is most easily confused with k 2 of the background image ( k 1 = k 2 ), we use them to construct a pair. However, using the confusion matrix to build the image-mixing pair is inferior to ours. The possible reason is that OTmix provides a more adaptive way to measure the importance of each background image for the foreground image at each training iteration. We provide more analytical results in Appendix 7.3.

5.5 The Learned Transport Plan for Image-mixing

Recall that our method adaptively learns a transport plan between a batch of foreground images and a batch of background images. Fig. 3 shows our learned transport plan on CIFARLT-10 with φ = 100 , whose list of class names is {airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck} from k = 1 to 10 . To facilitate the visualization, we set the size of B fore and B back as M = 10 , respectively. We observe that the learned transport plan can effectively capture the semantic correlations between foreground and background images. For example, the foreground image from the 'truck' class (from the 'ship' class) is highly related to the background image from the 'automobile' class (from the 'airplane' class). These correlations align with the confusion matrix in Fig. 6 of Appendix 7.3, however, OTmix is adaptive and specific to batches, while the confusion matrix is global and fixed. In addition to capturing the class-level similarity information, we

0.07 0.22 0.10 0.07 0.06 0.12 0.06 0.16 0.08 0.06

horse

0.00 0.00 0.00 0.00 0.000.000.750.240.00 0.00

dog

Foreground labels

0.24 0.01 0.15 0.24 0.14 0.02 0.01 0.00 0.18 0.01

ship

0.06 0.15 0.07 0.26 0.05 0.04 0.07 0.02 0.23 0.04

cat

0.03 0.24 0.03 0.02 0.05 0.05 0.09 0.350.03 0.11

frog

0.05 0.01 0.03 0.02 0.030.380.00 0.01 0.020.46

truck

0.11 0.03 0.14 0.17 0.17 0.12 0.09 0.02 0.10 0.05

0.02

20.53 0.02 0.02 0.04 0.04 0.04 0.15 0.03 0.11

automobile

deer

0.14 0.03 0.21 0.13 0.10 0.11 0.06 0.03 0.16 0.03

0.08 0.07 0.16 0.07 0.09 0.09 0.04 0.15 0.11 0.15

bird

airplane

airplane

airplane

Backgroundlabels

can find that different background images from the same class have different importance to the same foreground image, i.e., the sample-level similarity information. For instance, the transport

probabilities between the foreground dog image and two background cat images are 0 . 75 and 0 . 24 , respectively. We attribute that to our designed cost function based on the sample-level feature and class-level confusion matrix when learning the transport plan. Therefore, a learned transport plan is an effective and adaptive way to guide the image-mixing process for long-tailed classification. We further compare the image-mixing statistical results of OTmix and CMO in Appendix 7.5 at one epoch, i.e., the probability of each class in foreground distribution being mixed with each class in background distribution.

5.6 Visualization Results and Analysis

T-SNE Visualization To gain additional insight, we provide the T-SNE visualization of BALMS, BALMS+CMO, and BALMS+OTmix on CIFAR-LT-10 with φ = 100 in Fig. 4. Compared with BALMS+CMO, the majority class is more separable from the minority class2 in OTmix. Besides, the samples within the same class in our proposed method are usually closer than in CMO, i.e., small intra-class. It indicates that our proposed method can generate more effective mixed samples to improve the long-tailed classification performance.

(a) BALMS

(b) BALMS+CMO

(c) BALMS+OTmix

Majority Class

Minority Class 1

Minority Class1

Minority Class

Minority Class 1

Minority Class 2

Majority Class

Majority Class

Minority Class 2

Mixed Image Pairs Visualization To visually witness the intricate dynamics of the mixing process in our method, we provide the visualization results of image pairs for semantically meaningful mixed images on ImageNet-LT. As shown in Fig. 5, we plot the more semantically reasonable and relevant background image for each foreground image given the learned transport plan. For example, the semantic information of 'Grey fox' is semantically relevant to 'Dhole'. It visually reveals that our method can generate meaningful and advantageous samples, benefiting the balanced and imbalanced classification tasks. More visualization results are available in Appendix 7.5.

Linnet

Admiral

Lifeboat

Lemon

Taxi

Espresso

Drake

Ski

Greyfox

Background

Junco

Persiancat

Fly

Liner

GrannySmith

Car

Cup

BlackSwan

Snowmobile

Dhole

Mixed

6 Conclusion

We have introduced a novel adaptive image-mixing augmentation method based on OT for longtailed classification, with the goal of generating more semantically meaningful samples. We view the background images mainly from majority classes and the foreground images mostly from minority classes as two distributions, where we aim to minimize the OT distance between these two distributions. Moreover, we further design a combined cost function based on sample-level and label-level information. By viewing the learned transport plan as guidance to build a pair for image-mixing, we provide an effective way to weigh the background image for each foreground image. Our proposed OTmix has shown appealing properties that can be either combined with existing long-tailed methods or applied to balanced classifications. Experimental results validate that OTmix achieves competing performance on commonly long-tailed problems and balanced datasets.

References

Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. 'Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss'. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) . 2019.

Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. 'Balanced meta-softmax for long-tailed visual recognition'. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) . 2020.

Seulki Park, Jongin Lim, Younghan Jeon, and Jin Young Choi. 'Influence-balanced loss for imbalanced visual classification'. Proceedings of the IEEE/CVF International Conference on Computer Vision . 2021, pp. 735-744.

Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, et al. 'Disentangling label distribution for long-tailed visual recognition'. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 2021, pp. 6626-6636.

Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, et al. 'Meta-weight-net: Learning an explicit mapping for sample weighting'. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) . 2019.

Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. 'Learning to reweight examples for robust deep learning'. International conference on machine learning . 2018, pp. 4334-4343.

Dandan Guo, Zhuo Li, Meixi Zheng, He Zhao, et al. 'Learning to Re-weight Examples with Optimal Transport for Imbalanced Classification'. Advances in Neural Information Processing Systems . 2022, pp. 25517-25530.

Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, et al. 'Exploring the limits of weakly supervised pretraining'. Proceedings of the European conference on computer vision (ECCV) . 2018, pp. 181-196.

Jialian Wu, Liangchen Song, Qian Zhang, Ming Yang, and Junsong Yuan. 'Forestdet: Largevocabulary long-tailed object detection and instance segmentation'. IEEE Transactions on Multimedia 24 (2021), pp. 3693-3705.

Xinting Hu, Yi Jiang, Kaihua Tang, Jingyuan Chen, et al. 'Learning to segment the tail'. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 2020.

Junran Peng, Xingyuan Bu, Ming Sun, Zhaoxiang Zhang, et al. 'Large-scale object detection in the wild from imbalanced multi-labels'. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 2020, pp. 9709-9718.

Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. 'SMOTE: synthetic minority over-sampling technique'. Journal of artificial intelligence research 16 (2002), pp. 321-357.

Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. 'A systematic study of the class imbalance problem in convolutional neural networks'. Neural networks (2018), pp. 249-259.

Guo Haixiang, Li Yijing, Jennifer Shang, Gu Mingyun, et al. 'Learning from class-imbalanced data: Review of methods and applications'. Expert systems with applications (2017), pp. 220239.

Debashree Devi, Biswajit Purkayastha, et al. 'Redundancy-driven modified Tomek-link based undersampling: A solution to class imbalance'. Pattern Recognition Letters (2017), pp. 3-12.

Haibo He and Edwardo A Garcia. 'Learning from imbalanced data'. IEEE Transactions on knowledge and data engineering (2009), pp. 1263-1284.

Jaehyung Kim, Jongheon Jeong, and Jinwoo Shin. 'M2m: Imbalanced classification via major-to-minor translation'. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 2020, pp. 13896-13905.

Hongxin Wei, Lue Tao, Renchunzi Xie, Lei Feng, and Bo An. 'Open-sampling: Exploring out-of-distribution data for re-balancing long-tailed datasets'. International Conference on Machine Learning . 2022, pp. 23615-23630.

Yan Hong, Jianfu Zhang, Zhongyi Sun, and Ke Yan. 'Safa: Sample-adaptive feature augmentation for long-tailed image classification'. European Conference on Computer Vision . 2022.

Seulki Park, Youngkyu Hong, Byeongho Heo, Sangdoo Yun, and Jin Young Choi. 'The majority can help the minority: Context-rich minority oversampling for long-tailed classification'. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2022.

Hsin-Ping Chou, Shih-Chieh Chang, Jia-Yu Pan, Wei Wei, and Da-Cheng Juan. 'Remix: rebalanced mixup'. Proceedings of the European conference on computer vision (ECCV) . 2020, pp. 95-110.

Zhengzhuo Xu, Zenghao Chai, and Chun Yuan. 'Towards calibrated model for long-tailed visual recognition from prior perspective'. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) . 2021, pp. 7139-7152.

Peng Chu, Xiao Bian, Shaopeng Liu, and Haibin Ling. 'Feature space augmentation for long-tailed data'. European Conference on Computer Vision . 2020, pp. 694-710.

Shuang Li, Kaixiong Gong, Chi Harold Liu, Yulin Wang, et al. 'Metasaug: Meta semantic augmentation for long-tailed visual recognition'. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 2021, pp. 5212-5221.

Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. 'Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition'. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 2020, pp. 9719-9728.

Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, et al. 'Decoupling representation and classifier for long-tailed recognition'. Eighth International Conference on Learning Representations (ICLR) . 2020.

Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. 'Improving calibration for long-tailed recognition'. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 2021, pp. 16489-16498.

Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella Yu. 'Long-tailed Recognition by Routing Diverse Distribution-Aware Experts'. International Conference on Learning Representations . 2021.

Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. 'Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect'. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) . 2020.

AFM Shahab Uddin, Sirazam Monira, Wheemyung Shin, TaeChoong Chung, Sung-Ho Bae, et al. 'SaliencyMix: A Saliency Guided Data Augmentation Strategy for Better Regularization'. International Conference on Learning Representations . 2021.

Jie-Neng Chen, Shuyang Sun, Ju He, Philip HS Torr, et al. 'Transmix: Attend to mix for vision transformers'. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2022, pp. 12135-12144.

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. 'mixup: Beyond Empirical Risk Minimization'. International Conference on Learning Representations . 2018.

Zongbo Han, Zhipeng Liang, Fan Yang, Liu Liu, et al. 'Umix: Improving importance weighting for subpopulation shift via uncertainty-aware mixup'. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) . 2022.

Adrian Galdran, Gustavo Carneiro, and Miguel A González Ballester. 'Balanced-mixup for highly imbalanced medical image classification'. Medical Image Computing and Computer Assisted Intervention . 2021, pp. 323-333.

Gabriel Peyré and Marco Cuturi. 'Computational optimal transport'. Foundations and Trends in Machine Learning (2019), pp. 355-607.

Lu Yang, He Jiang, Qing Song, and Jun Guo. 'A survey on long-tailed visual recognition'. International Journal of Computer Vision (2022), pp. 1837-1872.

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, et al. 'Cutmix: Regularization strategy to train strong classifiers with localizable features'. Proceedings of the IEEE/CVF international conference on computer vision . 2019, pp. 6023-6032.

Shashanka Venkataramanan, Ewa Kijak, Laurent Amsaleg, and Yannis Avrithis. 'Alignmixup: Improving representations by interpolating aligned features'. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2022, pp. 19174-19183.

Jihao Liu, Boxiao Liu, Hang Zhou, Hongsheng Li, and Yu Liu. 'Tokenmix: Rethinking image mixing for data augmentation in vision transformers'. European Conference on Computer Vision . 2022, pp. 455-471.

Lingfeng Yang, Xiang Li, Borui Zhao, Renjie Song, and Jian Yang. 'Recursivemix: Mixed learning with history'. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) . 2022.

Zhi Hou, Baosheng Yu, and Dacheng Tao. 'Batchformer: Learning to explore sample relationships for robust representation learning'. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2022, pp. 7256-7266.

Anh Tuan Bui, Trung Le, Quan Hung Tran, He Zhao, and Dinh Phung. 'A Unified Wasserstein Distributional Robustness Framework for Adversarial Training'. International Conference on Learning Representations . 2021.

Irena Gao, Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. 'Out-ofDomain Robustness via Targeted Augmentations'. International Conference on Machine Learning . 2023.

Sara Beery, Yang Liu, Dan Morris, Jim Piavis, et al. 'Synthetic examples improve generalization for rare classes'. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision . 2020.

Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. 'Class-balanced loss based on effective number of samples'. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 2019, pp. 9268-9277.

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 'Focal loss for dense object detection'. Proceedings of the IEEE international conference on computer vision . 2017, pp. 2980-2988.

Aude Genevay, Gabriel Peyré, and Marco Cuturi. 'Learning generative models with sinkhorn divergences'. International Conference on Artificial Intelligence and Statistics . 2018, pp. 16081617.

Giorgio Patrini, Rianne Van den Berg, Patrick Forre, Marcello Carioni, et al. 'Sinkhorn autoencoders'. Uncertainty in Artificial Intelligence . 2020, pp. 733-743.

Viet Huynh, Dinh Phung, and He Zhao. 'Optimal transport for deep generative models: state of the art and research challenges'. International Joint Conference on Artificial Intelligence 2021 . 2021, pp. 4450-4457.

Tung-Long Vuong, Trung Le, He Zhao, Chuanxia Zheng, et al. 'Vector Quantized Wasserstein Auto-Encoder'. International Conference on Machine Learning . 2023.

Vy Vo, Trung Le, Long-Tung Vuong, He Zhao, et al. 'Learning Directed Graphical Models with Optimal Transport'. arXiv preprint arXiv:2305.15927 (2023).

Tuan Nguyen, Trung Le, He Zhao, Quan Hung Tran, et al. 'Most: Multi-source domain adaptation via optimal transport for student-teacher learning'. Uncertainty in Artificial Intelligence . 2021, pp. 225-235.

Dan dan Guo, Long Tian, Minghe Zhang, Mingyuan Zhou, and Hongyuan Zha. 'Learning prototype-oriented set representations for meta-learning'. International Conference on Learning Representations . 2022.

Dandan Guo, Long Tian, He Zhao, Mingyuan Zhou, and Hongyuan Zha. 'Adaptive Distribution Calibration for Few-Shot Learning with Hierarchical Optimal Transport'. Advances in Neural Information Processing Systems . 2022, pp. 6996-7010.

Long Tian, Jingyi Feng, Xiaoqiang Chai, Wenchao Chen, et al. 'Prototypes-oriented Transductive Few-shot Learning with Conditional Transport'. Proceedings of the IEEE/CVF International Conference on Computer Vision . 2023.

Miaoge Li, Dongsheng Wang, Xinyang Liu, Zequn Zeng, et al. 'PatchCT: Aligning Patch Set and Label Set with Conditional Transport for Multi-Label Image Classification'. IEEE/CVF International Conference on Computer Vision . 2023, pp. 15348-15358.

Dongsheng Wang, Miaoge Li, Xinyang Liu, MingSheng Xu, et al. 'Tuning Multi-mode Tokenlevel Prompt Alignment across Modalities'. Advances in Neural Information Processing Systems (2023).

Viet Huynh, He Zhao, and Dinh Phung. 'Otlda: A geometry-aware optimal transport approach for topic modeling'. Advances in Neural Information Processing Systems 33 (2020), pp. 1857318582.

He Zhao, Dinh Phung, Viet Huynh, Trung Le, and Wray Buntine. 'Neural Topic Model via Optimal Transport'. International Conference on Learning Representations . 2021.

Dongsheng Wang, Dandan Guo, He Zhao, Huangjie Zheng, et al. 'Representing Mixtures of Word Embeddings with Mixtures of Topic Embeddings'. International Conference on Learning Representations . 2022.

Dongsheng Wang, He Zhao, Dan Dan Guo, Xinyang Liu, et al. 'BAT-Chain: Bayesian-Aware Transport Chain for Topic Hierarchies Discovery' (2022).

Hanyu Peng, Mingming Sun, and Ping Li. 'Optimal transport for long-tailed recognition with learnable cost matrix'. International Conference on Learning Representations . 2022.

Yuguang Yan, Mingkui Tan, Yanwu Xu, Jiezhang Cao, et al. 'Oversampling for imbalanced data via optimal transport'. Proceedings of the AAAI Conference on Artificial Intelligence . 2019.

Jean-David Benamou, Guillaume Carlier, Marco Cuturi, Luca Nenna, and Gabriel Peyré. 'Iterative Bregman projections for regularized transportation problems'. SIAM Journal on Scientific Computing (2015), A1111-A1138.

Lenaic Chizat, Gabriel Peyré, Bernhard Schmitzer, and François-Xavier Vialard. 'Scaling algorithms for unbalanced optimal transport problems'. Mathematics of Computation (2018), pp. 2563-2609.

R Flamary, N Courty, D Tuia, and A Rakotomamonjy. 'Optimal transport for domain adaptation'. IEEE Trans. Pattern Anal. Mach. Intell (2017), pp. 1853-1865.

Aude Genevay, Marco Cuturi, Gabriel Peyré, and Francis Bach. 'Stochastic optimization for large-scale optimal transport'. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) . 2016.

Marco Cuturi. 'Sinkhorn distances: Lightspeed computation of optimal transport'. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) . 2013.

Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. 'Learning deep representation for imbalanced classification'. Proceedings of the IEEE conference on computer vision and pattern recognition . 2016, pp. 5375-5384.

Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. 'Learning to model the tail'. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) . 2017.

Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, et al. 'Exploring the limits of weakly supervised pretraining'. Proceedings of the European conference on computer vision (ECCV) . 2018, pp. 181-196.

Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. 'Class-balanced loss based on effective number of samples'. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 2019, pp. 9268-9277.

Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. 'Class-balanced loss based on effective number of samples'. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 2019, pp. 9268-9277.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, et al. 'Imagenet: A large-scale hierarchical image database'. IEEE conference on computer vision and pattern recognition . 2009.

Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, et al. 'The inaturalist species classification and detection dataset'. Proceedings of the IEEE conference on computer vision and pattern recognition . 2018, pp. 8769-8778.

Alex Krizhevsky, Geoffrey Hinton, et al. 'Learning multiple layers of features from tiny images'. Handbook of Systemic Autoimmune Diseases 1.4 (2009).

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, et al. 'Pytorch: An imperative style, high-performance deep learning library'. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) . 2019.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 'Deep residual learning for image recognition'. Proceedings of the IEEE conference on computer vision and pattern recognition . 2016, pp. 770-778.

Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, et al. 'Large-scale long-tailed recognition in an open world'. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 2019, pp. 2537-2546.

7 Appendix

7.1 Details of CutMix

To sample the binary mask M , we first sample the bounding box coordinates B = ( r x , r y , r w , r h ) indicating the cropping regions on x i and x j . The region B in x i is removed and filled in with the patch cropped from B of x j . In our experiments, we sample rectangular masks M whose aspect ratio is proportional to the original image. The box coordinates are uniformly sampled according to the:

making the cropped area ratio r w r h WH = 1 -λ . With the cropping region, the binary mask M ∈ { 0 , 1 } W × H is decided by filling with 0 within the bounding box B , otherwise 1.

7.2 More Comparison Results

In this section, we conduct comparison studies with mixup-based long-tailed methods, including Remix [21], Unimix [22], and CMO [20] under various loss functions. We summarize the results on CIFAR-LT-10 and CIFAR-LT-100 in Table 6 and list the results on ImageNet-LT and iNaturalist 2018 in Table 5. We can find that our OTmix surpasses related mixup-based methods with varying loss functions and enhances the imbalanced classification. Besides, OTmix with different losses produces different classification performances due to their characteristics, where OTmix with BALMS can achieve better or competing performance than OTmix with other loss functions. These results reveal the effectiveness of our proposed method when combined with other loss functions.

7.3 More Analytical Results

Confusion Matrix To verify whether our method improves the performance of minority classes, we show the confusion matrices of ERM-DRW, ERM-DRW+CMO, and ERM-DRW+OTmix on CIFAR-LT-10 with φ = 100 in Fig. 6. We can find that ERM-DRW suffers a severe performance drop in the minority classes even though it can almost accurately predict the samples in the majority classes. ERM-DRW+CMO can improve the accuracy of the minority classes, which coincides with the statement of CMO. OTmix further enhances the generalization of minority classes and

maintains performance in majority classes, which thus outperforms the strong baselines on the overall performance. In Fig.7, we plot the classification results for each class on CIFAR-LT-10 with φ = 100 , where we adopt the BALMS loss and LDAM loss, respectively. Compared with these baselines, OTmix provides a significant improvement in minority classes. We specifically note that the proposed method improves the accuracy over BALMS by 23% and over LDAM by 15% for the least frequent class 9 while degrading the accuracy for class 0 and class 1 by less than 2%. These results indicate that ours can achieve a more balanced classifier and ameliorate the generalization of minority classes.

Discussion of Hyper-parameters and Training Settings To analyze the effect of different hyperparameters and settings of OTmix, we conduct analytical experiments on CIFAR-LT-10 with φ = 100 . The hyper-parameters include ω , α , and r , respectively. ω in (7) is employed to manage the degree of combination of the confusion matrix and feature information in Fig. 8(a). The best performance is achieved when ω = 0.05, indicating the class-level cost based on the confusion matrix dominating the cost function. α affects the combined ratio in Cutmix in Appendix 7.1. As shown in Fig. 8(b), the larger the value of α is, the more complementary the distribution tends to be closer to a uniform distribution. The OTmix achieves an accuracy of 46.4% when α = 4.

ERM-DRW

ERM-DRW+CMO

ERM-DRW+OTmix

Q

0.970.010.010.000.000.000.000.000.000.00

0.930.010.020.010.000.000.000.010.010.01

0-

0.920.00 0.020.01 0.00 0.00 0.00 0.010.030.01

0.010.980.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01

1

0.010.970.000.000.000.000.000.000.000.01

0.010.960.000.000.000.000.010.000.000.02

0.8

0.07 0.010.820.04 0.03 0.02 0.02 0.00 0.00 0.00

0.040.000.81

0.040.040.020.030.010.000.00

label

3

0.04 0.01 0.070.77

0.03 0.06 0.02 0.01 0.00 0.00

0.030.080.020.010.00 0.00

3 0.02 0.00 0.040.710

0.03 0.13 0.04 0.02 0.00 0.00

0.6

40.03 0.00 0.06 0.050.810

0.020.010.020.000.00

e

50.04 0.00 0.05 0.19 0.040.66 0.01 0.01 0.00 0.00

0.010.020.000.00

5 0.01 0.00 0.02 0.12 0.03 0.790.01 0.02 0.00 0.00

0.4

60.03 0.00 0.07 0.11 0.04 0.010.730.00 0.00 0.00

0.010.000.050.090.030.010.81

0.010.00 0.00

0.000.000.00

70.07 0.00 0.03 0.09 0.08 0.05 0.000.67

0.000.00

7

0.020.000.030.060.100.050.010.730.000.00

70.01 0.00 0.02 0.04 0.08 0.07 0.010.770

0.000.00

8

0.360.06 0.01 0.01 0.00 0.00 0.01 0.000.540.01

8

0.180.030.010.020.000.000.010.000.740.01

0.03

0.2

6

0.150.27 0.00 0.02 0.00 0.00 0.00 0.00 0.020.53

0.070.180.000.020.00 0.000.00 0.01 0.020.70

6

0.04 0.10 0.01 0.01 0.00 0.01 0.010.00 0.030.80

0

1

2

3

4

5

6

7

8

6

0

1

2

3

4

5

6

7

8

9

0

1

2

3

4

5

6

7

8

6

0.0

Predictedlabel

Predictedlabel

Predictedlabel

(a) BALMS baseline

(b) LDAM baseline

9876543210

6

LDAM + OTmix

876543210

LDAM + CMO

LDAM

True label

True label

40

50

60

70

80

90

100

40

50

60

70

80

90

100

Accuracy (%)

Accuracy (%)

(a) Hyper-parameter

ω

(b) Hyper-parameter

α

(c) Hyper-parameter

r

(d) Training factor

µ

46.5

46.5

Accuracy (%)

46.0

Accuracy (%)

46.0

45.5

45.5

45.0

45.0

44.5

00.030.040.05 0.1 0.3 0.5 0.81

44.5

0.250.512345

6

46.5

46.5

46.3

46.0

Accuracy (%)

Accuracy (%)

46.1

45.9-

45.5

45.7

45.0

45.5

45.3

44.5

45.1

44.0

44.9

0.5

1

1.5

2

0.3

0.5

1

1.2

1.5

1.8

2

The hyper-parameter r controls the smoothness of q k , which further decides the sample weight of foreground distribution in (4). As shown in Fig. 8(c), the smoothness factor r = 1 achieves the best performance, which indicates the inverse class frequency. Recalling that we adopt y random ∼ Bernoulli ( t T ) to decide whether mixing the background and foreground pair with our OTmix, where we use µ to set y random ∼ Bernoulli ( t T ) µ and explore the effect of µ . As plotted in Fig. 8(d), ours with µ = 1 produces the best performance. That is to say, we will randomly mix the background and foreground images with a high probability during the first half of training epochs, and we are more likely to use OTmix to mix the background and foreground pairs in the latter half of epochs.

Mixed Fractions To examine the indispensability of the images unchanged in every batch, we conduct an additional experiment with the fraction of the mixed images in every batch on CIFAR-LT10 and CIFAR-LT-100 under different methods and fractions. We denote Fraction = Mixed Overall in every batch. From the results in Table 7, when the number of mixed samples decreases ( Fraction ↓ ), the performance of OTmix deteriorates significantly, which suggests that it is better to use mixed samples alone in every batch in our method.

Discussion of Confusion matrix Considering the confusion matrix played an important role in learning the cost function based class-level, we discuss the performance of our method and the computational cost of the normalized confusion matrix calculated on a balanced validation set, an imbalanced training set, and a small balanced subset sampled from imbalanced training set in Table 8. Meanwhile, the confusion matrix is represented as two states, fixed and adaptive. The former denotes where the confusion matrix remains unchanged in our approach and the latter changes dynamically in each epoch. From Table 8, we can draw a few observations: (1) With the same settings, the adaptive methods perform significantly better overall and few than the fixed methods. (2) Compared with the fixed balanced training setting D bal fixed , the fixed imbalanced training setting D im fixed is preferable by providing more sample information ( D im fixed ≫ D bal fixed ). However, large-scale samples can drastically increase the computational cost, making it difficult to implement adaptively. (3) Despite the time spent in the balanced training setting less, the balanced validation setting exhibits superior in terms of overall performance. To summarize, OTmix with the adaptively balanced validation setting enhances the suitability of OTmix for long-tailed classification.

7.4 Computational Cost

The optimal transport (OT) problem in our method between probability distributions is computed by the Sinkhorn algorithm [68], which introduces the entropic regularization term for fast computation. To compute the OT distance between n dimensional discrete distributions, the Sinkhorn algorithm requires the computational cost of O ( n 2 log ( n ) /ε 2 ) reach ε -accuracy. In our case, n corresponds to the batchsize, which is set to 128 in our experiments. We compare the computational cost of different methods on a Pentium PC with a single GTX 3060 GPU.

In addition, we also report the computational cost (s) per training epoch on long-tailed and balanced datasets, respectively. As shown in Table 9 and Table 10, mixup-based methods usually take more time than ERM. It is reasonable since mixup-based methods need to mix images. Besides, OTmix spends more time than CMO since we solve an OT problem to pair a background image and a foreground image. Still, introducing OTmix to existing mixup-based methods in balanced classification consumes more time. However, it is worth noticing that we only need to employ the OTmix during the late half phase of the training process. In summary, combining ours with others produces a better performance on long-tailed and balanced datasets with an acceptable cost.

7.5 More Visualization Results and Analysis

Statistical Results of Mixed Images To intuitively reveal that OTmix is more effective than CMO, we show the statistical results of the mixed images generated by CMO and OTmix on CIFAR-LT-10, respectively. Specifically, we summarize the 10 × 10 matrix m for the ten-class classification task in one training epoch, where element m ij denotes the number of pairs between the foreground images from the i -th class and the background images from the j -th class. As shown in Fig. 9(a), we can see that regardless of the foreground image from which class, CMO mainly mixes it with the background image from the majority classes. For example, the foreground images from the 'truck' will be mixed with the background images from the 'airplane' class, even if the truck is more similar to the automobile. However, Fig. 9(b) indicates that our proposed method builds more reasonable pairs by selecting the most relevant background image for each foreground image. For example, the 'horse' is more easily confused with 'deer' than 'airplane'. These results validate that ours can provide more reasonable generated samples than CMO.

Visualization Results of Mixed Image Pairs To gain a more intuitive insight into the dynamic changes within the mixing process of our method, we provide more visualization results of mixed image pairs on iNaturalist 2018 in Fig. 10. The foreground image and the selected background image commonly have significant semantic similarity. It suggests that OTmix has the capacity to generate reasonably mixed samples for long-tailed classification.

(a) CMO

(b) OTmix

493

308

168

117

75

43

27

18

10

9

500

495

282

191

130

59

39

24

17

8

4

airplane

Foreground labels

515

279

187

88

66

37

18

12

11

4

F400

bird

469

322

182

101

61

25

21

16

6

4

518

281

189

106

74

45

27

12

8

6

300

deer

511

314

172

116

53

43

25

13

6

2

6op

510

295

166

97

67

41

24

14

14

4

F200

frog

504

284

195

123

57

38

17

15

7

7

100

horse

487

310

171

98

69

29

20

8

7

8

ship

498

322

175

101

64

47

29

14

6

2

truck

9

deer

Background labels

34

66

598

0

1

17

86

138193

91

1200

102

80

0

0

13

30

313

315

209

193

airplane

llabels

277

22

0

4

187

15

518

102

49

35

1000

automobile

1

0

321

0

0

908

8

0

0

2

800

Foreground I

14

4

939

11

0

17

201

37

19

15

deer

0

0

0

1233

0

0

0

0

0

0

600

6op

0

0

1195

1

0

0

0

0

0

frog

F400

0

0

0

0

1198

43

0

0

0

0

horse

1272

0

0

0

0

0

0

0

0

0

F200

ship

0

1279

0

0

0

0

0

0

0

0

truck

0

deer

6op

airplane

Background labels

Foreground

Harrisii

Axis

RanaAurora

Achalarus

Lyciades

Imbricata

Barisia

Syncerus

Caffer

Flavoconia

Amanita

Abudefduf

Saxatilis

Latrodectus

Hesperus

Nereocystis

Luetkeana

Background

Leucurus

Antilocapra

Americana

Draytoni

Rana

Achlyodes

Pallida

Mississippiensis

Alligator

Muscaria

Amanita

Abudefduf

Sordidus

Leucauge

Palmaeformis

Postelsia

Mixed

7.6 Negative Societal Impacts and Limitations

This work develops a simple and effective image-mixing method for long-tailed learning, which has the potential to encourage researchers to derive new and better methods for the line of mixing images or long-tailed learning. However, if there is a sufficiently malicious or ill-informed choice of a long-tailed classification task or an image-mixing task, it may indirectly lead to a negative impact. Employing an imprecise or incorrect confusion matrix can mislead our method to build unsatisfactory pairs for image-mixing.

Peer reviews:
comments: SUMMARY:
This paper aim to solve long-tailed problem in the perspective of data augmentation. Specifically, the backgrounds of majority classes are mix with the foreground of minority classes. This work found out that randomly mixup images might face semantic misalignment, which impedes the effectiveness of this scheme. Therefore, this work use optimal transport to pair images with similar semantics. The paired images are mixed to form new training samples. On several benchmarks, this approach obtain good performance.

STRENGTHS:
This paper has following strengths:

- This paper identity a issue of former mixup-based long-tailed method that the mixed samples are semantically mismatched. This paper proposes a sound solution to tackle this issue.

- This paper is easy to follow.

- This method can be unifed with former LT methods as a plug-in module.

- This paper has a clear motivation.

WEAKNESSES:
This paper has following weaknesses:
 
- The performance gain on RIDE (3 experts) compared with CMO is limited.

- The contribution is limited since it's a updated version of CMO.

- The key point is to find semantical similar pairs, it's crucial to visualize some pair results.

- OT is to calculate optimal plan for aligning distributions. But the ideal solution for a minority sample is to find its cloest minority neighbor. Does using OT also lead to a suboptimal solution?

QUESTIONS:
The complexity of estimate $T$
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the interesting long-tailed classification problem, which is usually confronted by real-world machine learning tasks. In particular, this work extends the existing idea of mixing the background images from the majority class and foreground images from the minority class for imbalanced classification by incorporating class-level and sample-level information via the newly developed optimal transport-based image-mixing (OTmix) method. This paper has conducted extensive experiments on artificially crafted and real-world long-tailed datasets, with results demonstrating that OTmix outperforms various types of approaches designed for long-tailed classification, including augmentation-based mechanisms, and cost-sensitive methods, among many others.\n\nSTRENGTHS:\n1. The paper is well-written and presents information in a coherent manner, making it easy to comprehend.\n2. The introduced OTmix demonstrates the capability to enhance classification performance on imbalanced datasets. Additionally, it can serve as a valuable data augmentation technique for balanced datasets.\n3. The empirical evaluation in this study holds some validity. Extensive experiments have been conducted on diverse long-tailed datasets, and the results provide validation for the superiority of the proposed OTmix compared to state-of-the-art competitors. Furthermore, the ablation studies discussed in Section 5.2 illustrate the importance of each technical contribution made in this research.\n\nWEAKNESSES:\n### Major\n\n1. The technical contribution of this research is somewhat constrained. While this study expands on previous methods by incorporating sample-level and class-level information, its technical advancements largely rely on established image-mixing approaches. Hence, it can be considered more of an incremental contribution.\n2. This work would benefit from including a baseline comparison with well-known studies in long-tail classification tasks, such as Focal Loss [1] and Class-Balanced Loss [2]. This would provide a more comprehensive evaluation of the proposed approach.\n3. The rationale behind leveraging the confusion matrix to learn class-level information is insufficiently explained, making it challenging for readers to comprehend how the confusion matrix can be advantageous for long-tailed classification tasks. Further clarification is needed to elucidate the benefits and implications of utilizing the confusion matrix in this context.\n\n\n\n[1] Focal Loss for Dense Object Detection, https://arxiv.org/abs/1708.02002\n\n[2] Class-Balanced Loss Based on Effective Number of Samples, https://arxiv.org/abs/1901.05555\n\n### Minor\n\n1. In P4 L148, it is advisable to provide a concise overview of the proposed framework immediately following \"Fig 2.\" This will aid readers in quickly understanding the significant contributions of this research.\n2. Section 5.4 should also conduct explements for exploring the hyperparameter sensitivity of $w$ in Eq (7) and $\\lambda$ in Eq (9) on classification performance.\n\nQUESTIONS:\n1. In P3 L130, this paper mentions that \"which replaces the image region with a patch from another training image\". I wonder what is the patch size used in this paper. Do authors consider the effect of patch size on classification performance? To answer this question, I recommend conducting another ablation study by utilizing various batch sizes. e.g., 4x4, 8x8, or 16x16.\n2. How this study works during the inference stage? Should the data in the test set also mix the background and foreground images? If not, could you explain why image-mixing methods can improve the classification performance on the imbalanced dataset?\n\nLIMITATIONS:\nSee Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors tackle the notable challenge of learning from imbalanced data, proposing an adaptation to mixup-based or copy-paste-based data augmentation for rare categories that takes sample-level semantic similarity into account. Their proposed method uses optimal transport to underpin an adaptive mixup strategy, combining sample- and image-level information. They show this context-aware mixup strategy is useful as data augmentation for both long-tailed and balanced datasets.\n\nAfter reading the authors rebuttal and discussing with them, I will maintain my score of 6.\n\nSTRENGTHS:\nThe method is intuitively motivated, the implementation is reasonably simple, and the results are good and demonstrated on real-world imbalanced datasets such as iNaturalist. The authors further demonstrate improvement over vanilla mixup for balanced datasets like CIFAR.\n\nWEAKNESSES:\nThough they emphasize the semantic quality improvement of their method, it’s clear that this could perhaps be taken even further. For example, the method places pandas (a bear) on top of images of other bears, which is reasonable at a surface level. But these bears have drastically different habitats, at a more nuanced level it’s fundamentally unreasonable to see a panda and a black bear in the same habitat.\n\nPerformance is significantly worse on common categories in iNaturalist when using this method than ERM+CMO, and as common classes are the vast majority of what is seen in the real world, the drop in overall accuracy of this method when deployed on imbalanced test data might be difficult to overcome. \n\nNits:\n\nGrammar is often incorrect throughout, too many small errors to reasonably capture here\n\nIt would be good to reference https://arxiv.org/abs/2302.11861, which shows targeted copy-paste augmentation aligned with a novel test domain can improve out of domain robustness, a similar finding, as well as https://arxiv.org/abs/1904.05916 which shows rare-class performance improvement from synthetic data including a copy-paste baseline which includes context of night vs day backgrounds.\n\nQUESTIONS:\nBias and imbalance is often as related to where data is captured as it is to what is captured. How might bias in image backgrounds towards specific parts of the world, for example, impact the effectiveness of this type of approach?\n\nLIMITATIONS:\nLittle to no discussion of limitations or failure modes of this method, it would improve the quality of the work to understand where this helps and where it does not. Does performance for any rare category get worse?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
To tackle the data imbalance problem,  the authors suggest to manipulate the data distribution by introducing a new data augmentation mechanism.  This mechanism is a guided mix-up operation that uses backgrounds mostly from major classes and foregrounds from low frequency classes. To make the augmented images look more reasonnable, the authors suggest to match images based on a similarity at sample and/or class level. They formulate the  matching problem as a transport problem. The performance of the introduced augmentation is validated by several experiments on different benchmarks as well as balanced datasets.

STRENGTHS:
Although this paper introduces a simple extension of the mixup operation, the intuition behind it makes it an elegant solution. The formulation is technically sound and the claims are sufficiently supported with experiments. The paper is also well-written.

WEAKNESSES:
minor weakness: some typos that can be fixed easily

QUESTIONS:
Did the authors also consider to keep a fraction of the images unchanged in every batch? I am wondering if the trained networks will perform better in this case on real images than if they have been exclusively trained on mixed images

LIMITATIONS:
na
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06