SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\tilde {x} _ {i j ^ {\prime}} = \mathbf {M} \odot x _ {j} ^ {\prime} + (\mathbf {1} - \mathbf {M}) \odot x _ {i}, \quad \tilde {y} _ {i j ^ {\prime}} = \lambda y _ {j} ^ {\prime} + (1 - \lambda) y _ {i}. \tag {9}
$$
Training Details During the training process, we adopt a mini-batch setting to integrate our proposed OTmix with deep neural networks, where we learn  $\mathbf{T}$  and  $\theta$  alternatively. More specifically, at each iteration, we can sample a mini-batch  $\mathcal{B}_{\mathrm{fore}}$  from  $Q_{\mathrm{fore}}$  and another mini-batch  $\mathcal{B}_{\mathrm{back}}$  from  $P_{\mathrm{back}}$  to build  $\hat{Q}_{\mathrm{fore}}$  and  $\hat{P}_{\mathrm{back}}$ , both of which have  $M$  samples and  $M < N$ . In step (1), we can minimize the OT distance between  $\hat{Q}_{\mathrm{fore}}$  and  $\hat{P}_{\mathrm{back}}$  with (4) to learn the transport plan  $\mathbf{T}$ . In step (2), we can use the learned transport plan to select the most relevant background image  $x_j'$  for each foreground image  $x_i$  based on (8). In step (3), we can generate the mixed samples with (9) and minimize the classification loss to train the model  $g$  parameterized by  $\theta$ :
$$
L = \mathbb {E} _ {\left(x _ {i}, y _ {i}\right) \sim \mathcal {B} _ {\text {f o r e}}} \left[ \mathbb {E} _ {\left(x _ {j} ^ {\prime}, y _ {j} ^ {\prime}\right) \sim \mathcal {B} _ {\text {b a c k}}} [ \ell \left(y _ {i j ^ {\prime}}, g (\widetilde {x} _ {i j ^ {\prime}}, \theta)\right) ] \right]. \tag {10}
$$
Since the cost function relies on either the parameter  $\theta_{1}$  in the feature extractor or the parameter  $\theta$  in the whole model  $g$ , it might be inaccurate in the early training stage, resulting in an undesired transport plan matrix. Consequently, to avoid selecting the unsatisfactory background image for the foreground image and generating harmful mixed samples, it is more beneficial to randomly select  $x_{j}^{\prime}$  for  $x_{i}$  in the early stage. To this end, we adopt  $y_{\mathrm{random}} \sim \mathrm{Bernoulli}\left(\frac{t}{T}\right)$  to decide whether randomly mixing the background and foreground pair, where  $T$  is the number of training epochs and  $t$  indicates the current epoch. In the early stage, such as  $t \leq T/2$ , we can get  $y_{\mathrm{random}} = 0$  with a high probability, where we randomly mix the background and foreground images like CMO [20]; in the late stage, such as  $t > T/2$ , we are more likely to sample  $y_{\mathrm{random}} = 1$ , where we can learn  $\mathbf{T}$  with (4) and select the most relevant background image for each foreground image. Moving beyond CMO, we introduce a more adaptive and elegant image-mixing data augmentation method for long-tailed classification. We summarize our proposed method in Algorithm 1 and highlight steps (1), (2), and (3).
Using OTMix for Balanced Classifications In this case, our method can be viewed as a better alternative to other augmentation methods that mix images randomly [37, 30]. Applying OTMix in balanced classifications is similar to doing it in the imbalanced problems except that we construct both  $P_{\mathrm{back}}$  and  $Q_{\mathrm{fore}}$  as uniform distributions.
5.1 Experimental Settings and Implementation Details
Datasets Following [23, 25, 27, 20], we evaluate our method on long-tailed classification benchmark datasets: CIFAR-LT-10 [73], CIFAR-LT-100 [73], ImageNet-LT [74], and iNaturalist 2018 [75]. For
Algorithm 1 Adaptive Image-mixing Method with Optimal Transport.
Require: Training dataset  $\mathcal{D}$  , model  $g$  with parameter  $\theta$  , hyper-parameters  $\{\omega ,\alpha ,\beta ,\epsilon ,r\}$  .   
1: Build discrete distributions  $Q_{\mathrm{fore}}$  with (3) and  $P_{\mathrm{back}}$  with (2);   
2: for  $t = 1,2,\dots,T$  do   
3: Sample mini-batch  $\mathcal{B}_{\mathrm{fore}}$  from  $Q_{\mathrm{fore}}$  and  $\mathcal{B}_{\mathrm{back}}$  from  $P_{\mathrm{back}}$  ; Build  $\hat{Q}_{\mathrm{fore}}$  with  $\mathcal{B}_{\mathrm{fore}}$  and  $\hat{P}_{\mathrm{back}}$  with  $\mathcal{B}_{\mathrm{back}}$  .   
4: Sample  $y_{\mathrm{random}}\sim \mathrm{Bernoulli}(\frac{t}{T})$  .   
5: if  $y_{\mathrm{random}} = 0$  then   
6: Generate the mixed samples with (1), where we randomly choose  $x_{j}$  for  $x_{i}$  .   
7: else   
8: Step (1): Learn the transport plan T by minimizing OTe(  $\hat{Q}_{\mathrm{fore}},\hat{P}_{\mathrm{back}})$  with (4);   
9: Step (2): Select the most relevant background image  $x_j^\prime$  for foreground image  $x_{i}$  with (8);   
10: Step (3): Generate the mixed samples  $(\widetilde{x}_{ij^{\prime}},y_{ij^{\prime}})$  with (9);   
11: end if   
12: Update  $\theta^{(t + 1)}\gets \theta^{(t)} - \beta \nabla_{\theta}L$  with (10);   
13: end for
clarity, the imbalance factor is defined as the data point amount ratio between the most frequent and the least frequent classes, i.e.,  $\varphi = \frac{n_K}{n_1}$ . Among them, CIFAR-LT-10 (CIFAR-LT-100) are created from CIFAR-10 (CIFAR-100) [76] with  $\varphi = \{100, 50, 10\}$ , respectively. ImageNet-LT, containing 1,000 classes with  $\varphi = 256$ , is created from ImageNet [74]. Different from them, iNaturalist 2018 is a real-world and large-scale imbalanced dataset with 8,142 classes and  $\varphi = 500$ .
**Baseline Methods** We consider several baseline methods: (1) Empirical risk minimization (ERM), training on the cross-entropy loss; (2) Mixup-based augmentation methods: Remix [21], Unimix [22], CMO [20]; (3) Other augmentation methods: M2m [17], Open-sampling [18], FSA [23], SAFA [19]; (4) Losses designed for long-tailed classifications: LDAM[1], DRW [1], BALMS [2], IB [3], LADE [4]; (6) Other methods: BBN [25], RIDE [28], Decouple [26], MisLAS [27].
Implementation Details For all datasets, we use PyTorch [77] and SGD optimizer with momentum 0.9. Besides, we set  $\omega$  for defining cost function in (7) as  $0.05$ ,  $\epsilon$  for entropic constraint in (4) as  $0.01$ ,  $r$  for the smoothness in (3) as 1, and  $\alpha$  for sampling combination ratio as 4. For CIFAR-LT-10 and CIFAR-LT-100, we use ResNet-32 [78] as the backbone following [1] and use 240 epochs on a single GTX 2080Ti and set the initial learning rate as 0.1, which is decayed by 0.1, 0.1, and 0.01 at the 100th, 160th, and 200th epochs. For the ImageNet-LT, we employ ResNet-50 as the backbone following [20] and use 200 epochs on four GTX 2080Ti GPUs. The learning rate is initialized as 0.1 and decays by 0.1, 0.1, and 0.01 at the 40th, 80th, and 160th epochs. For the iNaturalist 2018, we use ResNet-50 as the backbone fol
lowing [1] and train 210 epochs on four Tesla A100 GPUs with an initial learning rate of 0.1, which is decayed by 0.1 in the 30th, 80th, 130th, and 180th epochs. Performances are mainly reported as the overall top-1 errors (\%). Following [79], we also report the error rates on three disjoint subsets: many-shot classes with more than 100 training samples, medium-shot classes with 20 to 100 samples, and few-shot classes with 20 samples.
Table 1: Top-1 errors (\%) of ResNet-32 on CIFAR-LT-10 and CIFAR-LT-100. " $\ddagger$ ": our reproduced results. " $\dagger$ ": results reported in the original paper.
CIFAR-LT-10	CIFAR-LT-100
100	50	10	100	50	10
ERM‡	26.9	22.9	12.9	63.2	56.3	43.4
LDAM‡[1]	26.3	21.6	13.4	61.1	56.0	44.4
ERM-DRW‡[1]	24.3	18.9	11.9	58.0	54.3	41.8
LDAM-DRW‡[1]	23.0	19.1	11.8	57.4	52.2	45.0
BALMS‡[2]	22.7	19.1	11.8	58.0	53.1	41.6
IB†[3]	21.7	18.3	11.7	55.0	51.1	42.0
LADE†[4]	-	-	-	54.6	49.5	38.3
BBN†[25]	20.2	17.8	11.7	57.4	53.0	40.9
RIDE (3 experts)‡[28]	18.4	16.0	13.7	51.4	48.6	40.2
MiSLAS†[27]	17.9	14.3	10.0	53.0	47.7	36.8
ERM + Open-sampling†[18]	22.4	18.2	10.6	59.7	55.2	41.9
ERM + M2m†[17]	21.7	-	12.1	57.1	-	41.8
LDAM-DRW + SAFA†[1]	19.5	16.4	11.1	54.0	50.0	40.9
ERM + Remix†[21]	24.6	-	11.8	58.1	-	40.6
ERM + UniMix†[22]	23.5	-	-	58.5	-	-
ERM + CMO[20]	25.0‡	18.6‡	11.5‡	56.1†	51.7†	40.5†
RIDE (3 experts) + CMO [20]	17.8‡	15.4‡	12.1‡	50.0†	47.0†	39.8†
ERM + OTmix	21.7	16.6	9.8	53.6	49.3	38.4
LDAM + OTmix	22.3	18.0	12.0	56.3	50.9	41.5
DRV + OTmix	16.9	13.8	9.4	52.0	47.4	37.3
LDAM-DRV + OTmix	18.2	16.0	11.8	52.0	47.6	41.0
BALMS + OTmix	16.0	13.5	9.8	53.2	47.7	37.7
RIDE (3 experts) + OTmix	17.3	14.8	11.3	49.3	46.2	39.2
Table 2: Top-1 errors (%) of ResNet-50 on ImageNet-LT and iNaturalist 2018. "★": results reported in CMO. "†": results reported in origin paper.
ImageNet-LT	iNaturalist 2018
ALL	Many	Medium	Few	ALL	Many	Medium	Few
ERM* [20]	58.4	36.0	66.2	94.2	39.0	26.1	36.5	44.5
ERM-DRW[1]	49.9*	38.3*	52.7*	71.2*	36.3†	-	-	-
LDAM-DRW* [1]	50.2	39.6	53.1	69.3	30.0	30.0	29.8	30.1
BALMS* [2]	49.0	39.1	51.2	67.9	30.0	30.0	29.8	30.1
IB† [3]	-	-	-	-	34.6	-	-	-
LADE* [4]	48.1	37.7	50.7	68.8	30.0	-	-	-
Decouple-cRT* [26]	52.7	41.2	56.0	73.9	31.8	26.8	31.2	33.9
Decouple-LWS* [26]	52.3	42.9	54.8	70.7	30.5	29.0	30.2	31.8
BBN* [25]	-	-	-	-	30.4	-	-	-
MiSLAS* [27]	47.3	-	-	-	28.4	26.8	27.6	29.6
RIDE (3 experts)* [28]	45.1	33.8	48.3	65.1	27.8	29.8	27.8	27.3
RIDE (4 experts)* [28]	44.6	33.8	47.7	63.5	29.1	29.1	27.6	26.9
ERM + M2m† [17]	57.8	-	-	-	-	-	-	-
Focal Loss+ FSA* [23]	-	-	-	-	34.1	-	-	-
LDAM-DRS + SAFA* [19]	-	-	-	-	30.2	-	-	-
ERM + Remix† [21]	58.3	-	-	-	38.7	-	-	-
ERM + CMO†	50.9	33.0	57.7	79.5	31.1	23.1	30.7	33.4
RIDE (3 experts) + CMO* [20]	43.8	33.6	46.1	64.4	27.2	31.3	27.4	26.9
ERM + OTmix	48.0	30.0	54.1	77.7	30.5	30.7	29.5	31.6
DRV + OTmix	46.6	33.0	51.0	69.6	28.9	29.4	28.1	29.6
LDAM-DRW + OTmix	47.5	37.2	48.9	71.8	30.4	31.5	29.8	30.6
BALMS + OTmix	44.4	36.0	47.6	57.3	28.5	28.9	28.0	29.2
RIDE (3 experts) + OTmix	42.7	40.6	43.5	55.9	27.0	28.7	27.2	26.2
5.2 Experiments on Long-tailed Classification
We conduct experiments on CIFAR-LT-10 and CIFAR-LT-100 of ResNet-32 in Table 1, where OTmix+ERM achieves better performance than the mixup-based augmentation methods with ERM loss, especially for CMO+ERM. It confirms the validity of selecting a suitable pair to mix. Besides, OTmix with ERM loss produces comparable performance to complex long-tailed classification methods, such as specially designed losses and other augmentation methods. Considering OTmix can also be combined with some existing long-tailed losses and model architectures, we further combine it with LDAM-DRW loss [1], BALMS loss [2], and also the network with multiple experts (RIDE) [28]. It is clear that OTmix equipped with BALMS loss, DRW loss, and RIDE outperforms most of the competing methods except on CIFAR-LT-10 with  $\varphi = 10$ , where it is only weaker than MisLAS. These results illustrate the superiority of the proposed OTmix.
We further perform experiments on large-scale imbalanced ImageNet-LT and iNaturalist 2018 datasets. Table 2 demonstrates that OTmix based on ERM loss can still produce a comparable performance when compared with competing long-tailed losses, related mixup-based, and other data augmentation methods. Similarly, combining ours with other losses or model architecture can overall outperform these related baselines on ImageNet-LT and iNaturalist 2018, where RIDE (3 experts) + OTmix performs best. Furthermore, our method can improve the performance of the medium and few classes, which is the goal of long-tailed methods. We compare OTmix with mixup-based augmentation methods under various loss functions in Appendix 7.2 and computational cost in Appendix 7.4.
5.3 Experiments on Balanced Classification
To validate whether our proposed OTmix can be used for the balanced classification task, we adopt the commonly-used CIFAR-10 [76] and CIFAR-100 [76]. Notably, our proposed OTmix can be flexibly combined with the classical mixup-based methods when mixing images, where we consider Mixup [32], Cutmix [37], and SaliencyMix [30]. We train the networks using the same details as on the CIFAR-LT. At each training iteration, we randomly sample two mini-batches, i.e.,
Table 3: Top-1 errors  $(\%)$  of our methods using ResNet-32 on balanced datasets.
Method	CIFAR-10	CIFAR-100
ResNet-32 (baseline)	11.8	34.2
+ Mixup [32]	9.8	32.9
+ OTmix (Mixup)	8.1	32.3
+ Cutmix [37]	8.6	31.6
+ OTmix (Cutmix)	5.8	28.0
+ SaliencyMix [30]	7.2	31.2
+ OTmix (SaliencyMix)	5.1	21.7
$\mathcal{B}_{\mathrm{fore}}$  and  $\mathcal{B}_{\mathrm{back}}$ , from the balanced training dataset. Table 3 lists the experimental results on balanced CIFAR datasets, where OTmix produces better performance. It indicates the benefit of building a better image pair to perform image-mixing data augmentation for the balanced classification task. Moreover, OTmix (SaliencyMix) achieves the best top-1 error of  $5.1\%$  and  $21.7\%$  on CIFAR-10 and CIFAR-100 datasets, respectively. These results show that OTmix is still effective on balanced
classification tasks and can be combined with some advanced mixup-based methods, proving its flexibility and availability.
5.4 Analytical Experiments
Here we implement the ablation study under various settings on CIFAR-LT-100 with  $\varphi = 100$  and summarize the results in Table 4. Our best-performing method uses the combined cost function with cosine similarity, sets the diagonal element in the confusion matrix  $\mathbf{F}$  as 0, and relies on the maximum value in the transport plan matrix to select the background image for each foreground image. To reveal the influence of the cost function, we denote the cost function by only using the sample-level cost and the class-level cost as "Sample-level cost only" and "Class-level cost only", respectively. Besides, we also consider using euclidean distance rather than cosine similarity to define the cost function called "Euclidean distance". Compared with ours, which combines the sample-level and class-level costs and uses cosine similarity, all these variants perform slightly worse, demonstrating the benefit of combining features and labels to compute the cost function and the effectiveness of cosine similarity.
To avoid mixing the background and foreground images from the same category, we set the diagonal element in  $\mathbf{F}$  as 0 and use the resultant  $\hat{\mathbf{F}}$  to compute the cost function. Here, we consider its variant by calculating the cost function using the original confusion matrix  $\mathbf{F}$ , denoted as "Keeping same category", which can be found to outperform by OTmix. It shows that avoiding mixing samples from the same class is beneficial, no matter from minority or majority classes. Besides, we
Table 4: Ablation study on CIFAR-LT-100.
Method	All	Many	Medium	Few
ERM + OTmix	53.6	27.7	53.2	83.3
Sample-level cost only	55.1	30.8	56.1	85.5
Class-level cost only	54.5	30.1	55.9	84.5
Euclidean distance	54.1	27.9	54.0	84.8
Keeping same category	54.9	29.0	55.1	85.0
Probability selection	55.0	28.0	55.7	85.6
Confusion matrix	57.1	29.4	59.5	86.8
adopt the multinomial distribution, denoted as "Probability selection", to select the corresponding background image for the  $i$ -th foreground image, whose probability vector can be formulated as  $\pmb{p} = \mathrm{softmax}(T_{i,1:M})$ . We observe that selecting background images according to probability distribution is inferior to using the maximum value, where the latter chooses the most relevant background image for each foreground image mainly from minority classes without randomness. It proves the necessity of using the maximum value in the transport plan to build a pair for mixing.
Furthermore, we use the confusion matrix itself ("Confusion matrix" in Table 4) to design the image-mixing pair instead of OT. The confusion matrix can be from a pre-trained model on the imbalanced training set with standard ERM loss. Specifically, after sampling the background and foreground images, if the class  $k_{1}$  of the foreground image is most easily confused with  $k_{2}$  of the background image  $(k_{1} \neq k_{2})$ , we use them to construct a pair. However, using the confusion matrix to build the image-mixing pair is inferior to ours. The possible reason is that OTmix provides a more adaptive way to measure the importance of each background image for the foreground image at each training iteration. We provide more analytical results in Appendix 7.3.
5.5 The Learned Transport Plan for Image-mixing
Recall that our method adaptively learns a transport plan between a batch of foreground images and a batch of background images. Fig. 3 shows our learned transport plan on CIFAR-LT-10 with  $\varphi = 100$ , whose list of class names is {airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck} from  $k = 1$  to 10. To facilitate the visualization, we set the size of  $\mathcal{B}_{\mathrm{fore}}$  and  $\mathcal{B}_{\mathrm{back}}$  as  $M = 10$ , respectively. We observe that the learned transport plan can effectively capture the semantic correlations between foreground and background images. For example, the foreground image from the "truck" class (from the "ship" class) is highly related to the background image from the "automobile" class (from the "airplane" class). These correlations align with the confusion matrix in Fig. 6 of Appendix 7.3, however, OTmix is adaptive and specific to batches, while the confusion matrix is global and fixed. In addition to capturing the class-level similarity information, we
Figure 3: Our learned transport plan on CIFAR-LT-10 with  $\varphi = 100$
can find that different background images from the same class have different importance to the same foreground image, i.e., the sample-level similarity information. For instance, the transport
probabilities between the foreground dog image and two background cat images are 0.75 and 0.24, respectively. We attribute that to our designed cost function based on the sample-level feature and class-level confusion matrix when learning the transport plan. Therefore, a learned transport plan is an effective and adaptive way to guide the image-mixing process for long-tailed classification. We further compare the image-mixing statistical results of OTmix and CMO in Appendix 7.5 at one epoch, i.e., the probability of each class in foreground distribution being mixed with each class in background distribution.
5.6 Visualization Results and Analysis
T-SNE Visualization To gain additional insight, we provide the T-SNE visualization of BALMS, BALMS+CMO, and BALMS+OTmix on CIFAR-LT-10 with  $\varphi = 100$  in Fig. 4. Compared with BALMS+CMO, the majority class is more separable from the minority class2 in OTmix. Besides, the samples within the same class in our proposed method are usually closer than in CMO, i.e., small intra-class. It indicates that our proposed method can generate more effective mixed samples to improve the long-tailed classification performance.
(a) BALMS
(b) BALMS+CMO
Figure 4: T-SNE visualization of BALMS, BALMS+CMO, and BALMS+OTmix on CIFAR-LT-10.
(c) BALMS+OTmix
Mixed Image Pairs Visualization To visually witness the intricate dynamics of the mixing process in our method, we provide the visualization results of image pairs for semantically meaningful mixed images on ImageNet-LT. As shown in Fig. 5, we plot the more semantically reasonable and relevant background image for each foreground image given the learned transport plan. For example, the semantic information of "Grey fox" is semantically relevant to "Dhole". It visually reveals that our method can generate meaningful and advantageous samples, benefiting the balanced and imbalanced classification tasks. More visualization results are available in Appendix 7.5.
Figure 5: The visualization results of image pairs for meaningful mixed images on ImageNet-LT.
We have introduced a novel adaptive image-mixing augmentation method based on OT for long-tailed classification, with the goal of generating more semantically meaningful samples. We view the background images mainly from majority classes and the foreground images mostly from minority classes as two distributions, where we aim to minimize the OT distance between these two distributions. Moreover, we further design a combined cost function based on sample-level and label-level information. By viewing the learned transport plan as guidance to build a pair for image-mixing, we provide an effective way to weigh the background image for each foreground image. Our proposed OTmix has shown appealing properties that can be either combined with existing long-tailed methods or applied to balanced classifications. Experimental results validate that OTmix achieves competing performance on commonly long-tailed problems and balanced datasets.
7.1 Details of CutMix
To sample the binary mask  $\mathbf{M}$ , we first sample the bounding box coordinates  $\mathbf{B} = (r_x, r_y, r_w, r_h)$  indicating the cropping regions on  $x_i$  and  $x_j$ . The region  $\mathbf{B}$  in  $x_i$  is removed and filled in with the patch cropped from  $\mathbf{B}$  of  $x_j$ . In our experiments, we sample rectangular masks  $\mathbf{M}$  whose aspect ratio is proportional to the original image. The box coordinates are uniformly sampled according to the:
$$
r _ {x} \sim \operatorname {U n i f} (0, W), \quad r _ {w} = W \sqrt {1 - \lambda}, \tag {11}
$$
$$
r _ {y} \sim \operatorname {U n i f} (0, H), \quad r _ {h} = H \sqrt {1 - \lambda}
$$
making the cropped area ratio  $\frac{r_w r_h}{W H} = 1 - \lambda$ . With the cropping region, the binary mask  $\mathbf{M} \in \{0,1\}^{W \times H}$  is decided by filling with 0 within the bounding box  $\mathbf{B}$ , otherwise 1.
7.2 More Comparison Results
In this section, we conduct comparison studies with mixup-based long-tailed methods, including Remix [21], Unimix [22], and CMO [20] under various loss functions. We summarize the results on CIFAR-LT-10 and CIFAR-LT-100 in Table 6 and list the results on ImageNet-LT and iNaturalist 2018 in Table 5. We can find that our OTmix surpasses related mixup-based methods with varying loss functions and enhances the imbalanced classification. Besides, OTmix with different losses produces different classification performances due to their characteristics, where OTmix with BALMS can achieve better or competing performance than OTmix with other loss functions. These results reveal the effectiveness of our proposed method when combined with other loss functions.
Table 5: Top-1 errors (%) of mixup-based long-tailed methods under various loss functions on ImageNet-LT and iNaturalist 2018. "★": results reported in CMO. "†": results reported in origin paper.
Method	ImageNet-LT	iNaturalist 2018
Mixup	Loss	ALL	Many	Medium	Few	ALL	Many	Medium	Few
None*	ERM	58.4	36.0	66.2	94.2	39.0	26.1	36.5	44.5
Remix†	ERM	58.3	-	-	-	38.7	-	-	-
CMO†	ERM	50.9	33.0	57.7	79.5	31.1	23.1	30.7	33.4
OTmix	ERM	48.0	30.0	54.1	77.7	30.5	30.7	29.5	31.6
None	ERM-DRW	49.9*	38.3*	52.7*	71.2*	36.3†	-	-	-
Remix†	ERM-DRW	-	-	-	-	29.5	-	-	-
CMO†	ERM-DRW	48.6	39.2	51.4	64.5	29.1	31.8	29.8	27.8
OTmix	ERM-DRW	46.6	33.0	51.0	69.6	28.9	29.4	28.1	29.6
None*	LDAM-DRW	50.2	39.6	53.1	69.3	30.0	30.0	29.8	30.1
CMO†	LDAM-DRW	48.9	38.0	52.6	69.2	30.9	24.7	30.5	32.7
OTmix	LDAM-DRW	47.5	37.2	48.9	71.8	30.4	31.5	29.8	30.6
None*	BALMS	49.0	39.1	51.2	67.9	30.0	30.0	29.8	30.1
CMO†	BALMS	47.7	38.0	50.9	63.3	29.1	31.2	30.0	27.7
OTmix	BALMS	44.4	36.0	47.6	57.3	28.5	28.9	28.0	29.2
7.3 More Analytical Results
Confusion Matrix To verify whether our method improves the performance of minority classes, we show the confusion matrices of ERM-DRW, ERM-DRW+CMO, and ERM-DRW+OTmix on CIFAR-LT-10 with  $\varphi = 100$  in Fig. 6. We can find that ERM-DRW suffers a severe performance drop in the minority classes even though it can almost accurately predict the samples in the majority classes. ERM-DRW+CMO can improve the accuracy of the minority classes, which coincides with the statement of CMO. OTmix further enhances the generalization of minority classes and