Learning to Re-weight Examples with Optimal Transport for Imbalanced Classification
Authors: Dan dan Guo
Zhuo Li
meixi zheng
He Zhao
Mingyuan Zhou
Hongyuan Zha
Venue: NeurIPS 2022 Accept (2022)
Keywords: imbalanced classification optimal transport  reweighting

Abstract: Imbalanced data pose challenges for deep learning based classification models. One of the most widely-used approaches for tackling imbalanced data is re-weighting, where training samples are associated with different weights in the loss function. Most of existing re-weighting approaches treat the example weights as the learnable parameter and optimize the weights on the meta set, entailing expensive bilevel optimization. In this paper, we propose a novel re-weighting method based on optimal transport (OT) from a distributional point of view. Specifically, we view the training set as an imbalanced distribution over its samples, which is transported by OT to a balanced distribution obtained from the meta set. The weights of the training samples are the probability mass of the imbalanced distribution and
learned by minimizing the OT distance between the two distributions. Compared with existing methods, our proposed one disengages the dependence of the weight learning on the concerned classifier at each iteration. Experiments on image, text and point cloud datasets demonstrate that our proposed re-weighting method has excellent performance, achieving state-of-the-art results in many cases and
providing a promising tool for addressing the imbalanced classification issue. The code has been made available at
https://github.com/DandanGuo1993/reweight-imbalance-classification-with-OT.

Learning to Re-weight Examples with Optimal Transport for Imbalanced Classification
Dandan Guo 1,2, Zhuo Li Meixi Zheng', He Zhao Mingyuan Zhou', Hongyuan Zha
'School of Data Science, The Chinese University of Hong Kong, Shenzhen
2 Institute of Robotics and Intelligent Manufacturing
Schooo of Science and Engineering, The Chinese University of Hong Kong, Shenzhen Shenzhen Research Institute of Big Data
Xidian University $CSIRO's s Data61 "The University of Texas at Austin
Shenzhen Institute of Artificial Intelligence and Robotics for Society guodandane cuhk edu. on 221 908801 likk, cuhk. edu. cuhk.edu... en meix i zherg 1100 63. com he. zhaoθ eee.org mi minguaan. zhou@mccombs utexas edu zhahy@cuhk eddu.
Abstract
Imbalanced data pose challenges for deep learning based classification mod- els. One of the most widely-used approaches for tackling imbalanced data is re-weighting, where training samples are associated with different weights in the loss function. Most of existing re-weighting approaches treat the example weights as the learnable parameter and optimize the weights on the meta set, entailing expensive bilevel optimization. In this paper, we propose novel re- weighting method based on optimal transport (OT) from a distributional point imbalanced distribution
of view. Specifically, we view the training set over its samples, which is transported by OT to balanced distribution obtained from the meta set. The weights of the training samples are the probability mass the imbalanced distribution and learned by minimizing the OT distance be- tween the two distributions. Compared with existing methods, disengages the dependence of the weight learning on the concerned classifier each iteration. Experiments image, text and point cloud datasets demon- strate that our proposed re- weighting method has excellent performance, achieving sate-of-the-mee art results in many cases and providing promising tool for address- ing the imbalanced classification issue. The code has been made available at Oith-OT
proposed one
https://gituub.com/aaddaa (aa /
Introduction
Deep neural networks (DNNs) have achieved remarkable success various applications, which is undoubtedly inseparable from the high high-quality large-scale datasets. Usually, the number of samples for each class in these datasets are manually selected resulting balanced datasets. However, most real-world datasets are imbalanced, such as a few classes (a.k.a. head or majority class) occupy most of the data while most classes ka. tail or minority class) have a few samples. A model trained on the imbalanced training set but without considering such class imbalance would be significantly dominated by those majority classes, and thus underperform on a balanced test dataset. This can also be known the long-tailes problem and exists in many domains, such as text classification [ ( object detection and image classification 4[6).
There are rich research lines to solve the imbalance problem, including sampligg 100 class-level or instance-level re- weightntg 2 18, meta-learning 15 16
two-stage methods
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

and post-hoe correction 20,211, Inspired by [2) re-weighting strategies can be roughly grouped into empirical re-weighting and automatic re-weighting, The former aims to design weights manually with the major insight that the minority class example will be assigned a larger weight value than that of the majority class [2. However, manually setting weights can be less adaptive to different datasets (2). The latter aims to assign adaptive weights to the examples through learning mechanisms M 2, 4 [6] As the representative automatic re-weighting method, L2RW [IS] optimizes the weight vector as a learnable parameter with unbiased meta set (i.e., validation set). Although L2RW and its followers have received widespread attention, most of them may be limited to optimizing the weights by the classification loss on the meta set: The gradient of weights is usually coupled with the to- to-be-learned classifier at each training iteration. Since classifier is the major concern in imbalanced issue [6], the dependence of weights on classifier at training stage may lead to inaccurate learning of the weights.
46 17
This paper develops a novel automatic re- re-weighting method for imbalanced classification based on optimal transport (OT). As discussed by Jamal et al. 4, the major challenge for imbalanced classification is essentially the mismatch between the imbalanced training dataset (seen by machine learning model) and the balanced test set (used to test the learned model). To this end, we aim to view the learning of the weight vector as the distribution approximation problem. We adopt the two-stage learning manner motivated by ( where stage I and stage 2 focus on learning the feature extractor with the standard cross-entropp loss and the classifier with our proposed method, respectively. Specifically, we represent the imbalanced training set as a discrete empirical distribution P over all samples within it and view the to-be-lcameed weight vector as its probability measure. Then we represent the balanced meta set as a discrete empirical distribution Q over all samples within it (in the same space with P) ). which has uniform probability measure for being balanced. Therefore, the learning of a weight vector can be formulated as the process of learning the distribution P to be close to the balanced distribution Q as possible, process facilitated by leveraging the OT distance (22). Notably, the cost function plays a paramount role when learning the transport plan for OT, where we use the features and ground-truth labels of samples design it. Due the flexibility of our method, we can also learn an explicit weight net directly from data like [16 23 but with different structure, optimized by OT loss instead of the classification loss on the meta set. Generally, at each training iteration at stage 2, we minimize the OT loss to learn the weight vector (or weight net) for the current mini-batch, which is further used to re-weight the training loss for optimizing the model. As we can see, the gradient of weights only relies on the OT loss and thus is independent of the classifier. More importantly, our proposed method is robust to the distribution Q. To save the memory consumption, we introduce the prototype-oriente OT loss by building a new distribution Q based on prototypes instead of samples (one prototype for each class). More importantly, our proposed method can achieve reasonably good performance even if we randomly select a mini-batch from all prototypes to build Q, making our method applicable to datasets with a large number of classes.
We summarize our main contributions as follows: (1) We formulate the learning of weight vector or weight net as the distribution approximation problem by minimizing the statistical distance between learned distribution over samples from imbalanced training set and another balanced distribution over samples from the meta set. (2) We leverage the OT distance between the distributions to guide the learning of weight vector or weight net. (3) We apply our method to imbalanced classification tasks including image, text and point cloud. Experiments demonstrate that introducing the OT loss to learn the example weights can produce effective and efficient classification performance.
2 Related Work
Empirical weightttning A classic empirical re- re-weighting scheme is to provide the examples of each class with the same weight, such as inverse class frequency [4] It has been further improved by the class-balanced loss [13. which calculates the effective number of examples as class frequency. Focal Loss [2] uses the predicted probability to calculate higher weights for the hard examples and dynamically adjust the weights. LDAM-DRW [17] designs label-distributiin--waaa and adopts a deferred class-level re-weighting method (i.c., inverse class frequency).
loss function
Automatic Re-weighting The automatic re-weighting methods learn the weights with learning mechanisms. 1.2RW [15] adopts a meta-learning manner to learn the example weights, which are optimized by the classification loss on the balanced meta set. Hu et al. [I] further improve L2RW
2

---

by iteratively optimizing weights instead of re-estimation at each iteration. Meta-weight-net [6 aims to learn an explicit weight net directly from data and optimize it by a meta-learning manner. Meta-class- Meta-class-weight [4] defines the weight for each example as the combination of class-level weight (estimated by Cui et al. [3] and instance-level weight, optimized with a meta-learning approach similar to 1.2RW. Influence-
loss (IB) is proposed to [8] re-weight samples by the magnitude of the gradient. Recently, Liu et al. [2] propose to update the weights and model under a constraint. Our method belongs to automatic re-weighting group, and the idea of building
explicit weight net similar to Shu et al. [6]. However, the major difference is that we bypass the classification loss on the meta set and use OT to learn the weights from the view of distribution approximation, disengaging the dependence of the weight learning on the concerned classifier at each iteration.
Meta Learning and Two-stage Learning Recently, researchers have proposed to tackle the imbal- ance issue with meta-learning, which can be applied to build a Balanced Meta-Softmax (BALMS) 19] learn weights 4 15 [6] or transformed semantic directions for augmenting the minority classes in MetaS [5) Two-stage methods, where the first stage and second stage focus on representation learning and classifier learning, respectively. have been proved effective for solving the imbalanced
issue 5 6 16 (4). BBN (25 unifies two stages with a specific cumulative learning strategy.
Optimal Transport Recently, OT has been used to solve the regression problem under the covariate shift (6),, unsupervised domain adaption 27 28 including sample-level class-level or domain-leve weight vector. Although they also adopt the re- re-weighting strategy and OT distance, they are distinct form ours in terms of task and technical detail. Also, the dynamic importance weighting which adopts MMD to re- re-weight samples for label-noise and class-prior-shift tasks [29) is also different from ours, where we provide a more flexible way for learning the weights of samples and disengage the dependence of the weight learning on the concerned classifier at each iteration. To the best of our knowledge, the works that solve imbalanced classification problem with OT are still very limited. An oversampling method via OT (OTOS) (30 aims to make synthetic samples follow similar distribution to that of minority class samples. However, ours is a novel re-weighting method based on OT, without augmenting samples. Another recent work is Optimal Transport via Linear Mapping (OTLM) 21). which performs the post-hoc correction from the OT perspective and proposes a linear mapping to replace the original exact cost matrix in OT problem. Different from OTLM that belongs to the post-hoc correction group and aims to learn refined prediction matrix, ours falls into the training-aware group and aims to re-weight the training classification loss.
3 Background
Imbalanced Classification Consider a training set Dramm = [(x;; y)) where (z. y) is the input and target pair, x, the i-th sample, y: E 1)K the one-hot associated label vector over K classes, and N the number of the entire training data. Besides, consider small balanced meta set Dimeta (Zz, yj)
where M is the amount of total samples and M < N. Denote the model parameterized with O as f(z 8), where O is usually optimized by empirical risk minimization over the training set, i.e., 0" = arg ming N EN (yis ff (z;: 0)). For notational convenience, we denote Himin = l (yr,, f (2r;; 0)) to represent the training loss function of pair (Ia,, ys). However, the model trained by this method will prefer the majority class if the training dataset is imbalanced.
y;))ş=1 M
Learning to Re- Reigtt Examples To solve the imbalanced issue, kind of re-weighting methods is to treat the weights as the learnable parameter and Iearn a fair model to the minority and the majority classes by optimizing the weighted training loss. At each training iteration, the model is updated by
.V
0" (w) arg min
W.rarain (0),
where w= (W1, (W....., (WT) is the weight vector (usually with simplex constraint) of all training examples. Then the optimal w is obtained by making the model parameter 0* ('w) from Eq. (I) minimize the classification loss on balanced meta set, formulated as
M Σ
arg min
5=1 myen (0" (w)),
where uneett is the loss function of pair (Iz, 3j) from meta set and the updated w is used to ameliorate the model. Generally, model O consists of two key components, feature extractor and classifier, where
3

---

the classifier has been proved to be the major concerning part in imbalanced issue [6]. However, the gradient of weights in Eq. 2 always depends on the co-be-concerned classifier each training iteration, which may result in inaccurate learning of the weights. Most automatic re-weighting methods learn the weight vectors or weight-related parameters (c.g., weight net) following this line;
see more details from the previous works 4 15 [6].
Optimal Transport Theory OT has been widely used calculate the cost of transporting one probability measure to another in various machine learning problems, such as generative models [I text analysis 32 331, adversarial robustness [3],, and meta learning [35 36]. Among the rich theory of OT, this work presents brief introduction to OT for discrete distributions; see Peyré and Cuturi [22] for more details. Consider p = Σi=1 agdz, and q = Σ byōy, as two probability distributions, where zj and y, live in the arbitrary same space and is the Dirac function. Then, we can denote E A and b E Am as the probability simplex of R" and R", respectively. The OT distance between p and g can be expressed as:
OT(p,)) g) min (T, C)2 Tell(p.))
(3)
where ( is the Frobenius dot-product and C E R2 is the transport cost matrix constructed
by C.j = Ca.,, yj), The transport probability matrix T E 1250 which satisfies I[(p, q) = (T ET Ti = bj. دド= Ti = a.j. is learned by minimizing OT(p,g). Directly optimizing Eq. (3) often comes the cost of heavy computational demands, and OT with entropic regularization is introduced to allow the optimization at small computational cost sufficient smoothness 337
4
Re-weighting Method with Optimal Transport
This work views a training to-be-learned distribution, whose probability measure is set as learnable weight vector w. We use OT distance to optimize wo for re-weighting the training loss.
4.1 Main Objective
Given the imbalanced training set Diriin we can represent it as empirical distribution over N pairs, triin has the sample probability w/ (i.c., the weight), defined as:
where each pair (zz, y4) )
P(w) Σ im1 w;;(s....))
(4)
where (I,, y4 )rmin is the i-th pair from the training set and the learnable weight vector w of all training examples means probability simplex of R.N. Since the meta set Pmota is balanced for all classes and closely related with the training set, it is reasonable to assume that meta set has already achieved the balanced data distribution that the training aims to approximate. For meta set, thus sample each pair from it with equal probability and present with an empirical distribution Q:
M
Q=
M 6
(5)
where (zy, v1) ) is the j-th pair from the meta set. To learn w, different from most automatic re- weighting methods, which minimize the classification loss on the meta set, we aim to enforce the to-be-learned distribution P(w) to stay close to the balanced distribution Q. Here, we explore the re- ee-weighting method by adopting the OT distance between P (w and Q:
min (T(P))) Q) def. min min TEM((((m)..
(T, C),
(6)
where cost matrix C' R.N. N is described below and transport probability matrix TER should 20
satisfy I1(P(w), Q) = T
ΣN
i= Tiy = 1/M Te 1 w.
4.2
Cost Function
For notation convenience, we reformulate the model as f(x 0) = (x; where fi param- eterized with 0, denotes the representation learning part before the classifier, and fa parameterized

---

with 02 denotes the classifier. Intuitively, the cost Ci measures the distance between pair i in training set and pair in meta set, which can be flexibly defined in different ways. We explore few conceptually intuitive options of Cij. although other reasonable choices can also be used.
Label-aware Cost As the first option, we can define Cit with the ground-truth labels of two samples:
Cj d (y/ (y
where d² (:) also denotes a distance measure, and y yo are the ground-truth label vectors of the two samples, respectively. Intuitively, if we use the euclidean distance, then C is 0- matrix (we can transfer the non-zero constant to 1), i.e., Cty= =0 if z:!iin and = are from the same class, and Cay= = otherwise. Now the OT loss is influenced by neither feature extractor 0 nor classifier 02.
Feature-aware Cost Besides, we can define Cij purely based on the features of samples:
Ci = d (2) ; x
(8)
where 2!iin = fi (2itrinn 01) E R and = = fa (1rrjto,,, 01) E E denote the E-dimensional representation of x.! and motta respectively. a (1,+) denotes any commonly used distance measure and we empirically find the cosine distance is a good choice. It is easy to see that if aq and z. 2:mictass features are close, their cost is small. Here the OT loss is influenced by the feature extractor 01.
Combined Cost Finally, we can use both features and labels to define Cij. denoted as
C25 c Fea [z]
(2) ² == + all"" (y! y.
(9)
Intuitively, Ci will be small if two samples have the same label and similar features. Empirically,
we find that using the d cosine(-. and euclidean distance for ddaا gives better performance. Interestingly, given the feature-awant cost (8) or label-aware cost (7). the learned weight vector can be interpreted the instance-level or class-level re- weighting method, respectively. The weight vector learned from the combined cost can be interpreted as the combination of class-level and instance-level weights, although no specialized design for two-component weights like previous see Fig.
4.3 Learn the Weight Vector
Given the defined cost function, we adopt the entropy regularized OT loss 37 to learn the weight vector. We thus rewrite (6 as the following optimization problem:
min Lot = (C, T (w)) subject to (w))= arg min w
(T, - - (H())
(10)
TEM([(w).)) Q)
where À > 0 is hyper-parameter for the entropic constraint HTT)= TiylnTij. Note that (10) provides us a new perspective to interpret the relationship between to and T, where w is the parameter of the leader problem and T is the parameter of the follower problem, which is of the lower priority. Accordingly, when we minimize (00) with respect to w using gradient descent, we should differentiate through T. Below we investigate the following two ways to optimize the weight vector.
Optimizing w directly Specifically, at each training iteration, we define P(w) with current w, use the Sinkhorn algorithm (37 compute OT loss, then optimize w by w" arg min Lor-
Amortizing the learning w We also provide an alternative method by constructing an explicit weight net to output the example weights, whose structure can be designed flexibly. For example, we can build the following weight net and take the sample features as input:
softmax (8) sf Watt tanh (Wuzz! U2 maain
(11)
where s² is the i-th element of E N Watt E IRIXA and W uz E AA are the learned parameters (we omit the bias for convenience). denoted as Ω {watts W ]. Denote S(2; (2) as the weight net parameterized by Z which can be optimized by π arg ming Lor.
5 Overall Algorithm and Implementations
To integrate our proposed method with deep learning frameworks, we adopt stochastic setting, i.e., mini-batch setting at each iteration. Following 4. 5). we adopt two-stage learning, where
5

---

Algorithm 1 Workflow about our re-weighting method for optimizing 0 and w.
Require: Datasets Dusin Dmeea initial model parameter 0 and weight vector, hyper-parameters 3, for = 1000-,, 2.., t1 do
Sample a mini-batch B from the training set Darain
Update ((++1) t g(t) a VeLB where Lg [p] Eieb& (u. (2):
end for
for 1 l +1, -1,.,,/ 41 do
Sample a mini-batch B from the training set Dirain
Step
(a):
Update
((++)) (w)()) (m)())
O VVoLB
where Ln
[8| EiEB w! f
24: (())
Use Denca to build Q in (12) and with w' to build P(w*)
44:
Step (b): Compute Lor
0 (++1)
(w'), wil)) with cost
Optimize w(++1) + w(()
V WwLon Lor 0 (1+1) ( (w"),w(())
Step (c): Update ((++1) <- g(0) VOLB where LB [m] EieB w; ((+1)g (yr. ( )
end for
stage I trains the model f(0) by the standard cross-entropy loss on the imbalanced training set and stage aims learn the weight vector and meanwhile continue to update the model ((O). Generally, at stage 2, calculating the optimal 0 and w requires two nested loops of optimization, which is cost-expensive. Motivated by Hu et al. [T] we optimize 0 and w alternatively, corresponding to (I) and (IO) respectively, where w is maintained and updated throughout the training, so that re-estimation from scratch can be avoided in each iteration. The implementation process of our proposed method with w optimized directly is shown in Algorithm I where the key steps are highlighted Step (a), (b), and (c). Specifically, at each training iteration f,, in Step (a), we have
0 (++1) (w") (0) (1+1) (w=), 02 (1++1) (w")) and the step size for 0: Step (b), as the cost function (w"),
(w"))
based on features is related with 0 (++1), o(++'((w") the OT loss relies on 0 (t++1) (w*), and |3 is the step size
for w: in Step (c), we ameliorate model parameters ((++1), We defer the learning of and Ω for the amortized learning of w.
Discussion From Step (b), we find the gradient of w is unrelated to classifier 0₂ regardless which cost function we choose. If we use the label-aware cost freeze the feature extractor parameterized by 0, which is trained in the first stage, the OT loss Step (b) can further reduced as Lor (w²), where only need Steps (b)-(c) each iteration. This is different from most of automatic re- weighting methods, where the gradient of w is always related with the to-be-learned model (61, 02) or classifier O2 (when freezing 01) for minimizing the classification loss on meta
Prototype-oriented OT loss (POT) Recall that we represent balanced meta set with M samples as distribution Q in 5). where M/K K is the number of data in each class and usually larger than 1. Computing the OT loss requires to learn a B x M -dimensioona transport matrix at each iteration. To improve the efficiency of algorithm, average all samples from each class in the meta set to achieve its prototype and propose new Q distribution over K prototypes:
K Q - -
K
M/K
- (EE Jam
meta [kj
(12)
k- K
M
where POT loss only needs a B x K- K-dimensional transport matrix. Due to the robustness of our method to Q when dealing with large number of classes, we can randomly sample mini-batch from K prototypes at each iteration to build Q.
6 Experiments
We conduct extensive experiments to validate the effectiveness of our proposed method on text, image, and point cloud imbalanced classification tasks. Notably, different from the imbalanced image

---

and point cloud classification, we find that optimizing the weight net is better than optimizing the weight vector directly the text classification. Therefore, we optimize the weight vector for the image and point cloud cases and build a weight net for text case. Unless specified otherwise, we adopt the combined cost and set the hyper-parameter for the entropic constraint as À = 0.1 and the maximum iteration number in the Sinkhorn algorithm as 200. We define the imbalance factor (IF) of a dataset as the data point amount ratio between the largest and smallest classes.
6.1 Experiments on Imbalanced Image Classification
Datasets and Baselines We evaluate our method on CIFAR-LT-IO. CIFAR-LT- 100, ImageNet-LT and Places-LT. We create CIFAR-LT-I0 (CIFAR-LT-100) from CIFAR-I0 (CIFAR- (00) by downsampling samples per class with IF {200, 100, 50, 20)
B3. ImageNet-LT is built from the classic ImageNet with 1000 classes 33 and IF= 1280/5 5 [5] 24]. Places-LLT' is created from Places-2
[40] with 365 classes and =F=4980/4
4 24 We randomly select 10 training images per class as
meta set [5] see more details in Appendix
B We consider the following baselines: (1) Cross-entropy
(CE), the model trained on the imbalanced training set with CE loss. (2) Empirical re- re-weighting
methods, like Focal loss [12. Class-balanced (CB) loss [3] and LDAM-DRW [17]. (3) Automatic
re-weighting methods, including L2RW [15]. IB [18] Meta-Weight-Net [66 and Meta-class-weight
[4] (4) Meta-lcarning methods, including MetaSAug [5] and above methods of [5] 16 19). (5)
Two-stage methods, such as OLTR (24). cRT 66. LWS ( BBN 225 and methods of 5..
Experimental details and results on CIFAR-LT For a fair comparison, we use ResNet-32 [4] as the backbone CIFAR-LT-IO and CIFAR-LT- 100. Following Li ct al. [5], at stage 1, we use 200 epochs, set the learning rate a of as 0.1, which is decayed by 1e-2 at the 160th and 180th epochs. At stage 2, we 40 epochs, set a as 2e 2 and learning rate 3 of weights as 1e- We the SGD optimizer with momentum 0.9, weight decay 5e -4 and set the batch size as 16. We list the recognition results of different methods on CIFAR-LT-IO and CIFAR-LT-I00 with different imbalance factors in Table We report the average result of 5 random experiments without standard deviation which is of small scale(e.g.. 2e- We can see that our re-weighting method outperforms CE training by large margin and performs better than the empirical or automatic re- re-weighting methods. Remarkably, our proposed method outperforms competing MetaSAug that conducts meta semantic augmentation approach to learn appropriate class-wise covariance matrices when IF is 200, 100 and 50. Importantly, as the training data becomes more imbalanced, our method more advantageous. Even though our proposed method is inferior to MetaS when the dataset is less imbalanced (IF=20) it can still achieve competing results and surpasses related re- weighting methods. This suggests that our proposed method can be used to enhance the imbalanced classification, without the requirement of designing complicated models or augmenting samples on purpose.
To more comprehensively understand our method, we provide a series of ablation studies on CIFAR- LT-100 with IF = 200 in Table 2 Firstly, to explore the impact of cost function, we use different cost functions for the OT loss. We can see that the combined cost performs better than label-aware cost and feature-aware cost, confirming the validity of combining features and labels to define cost. Besides, using either label-aware or feature- indicating the usefulness of OT loss in the imbalanced issue. Secondly, to explore the robustness of the meta distribution Q we adopt three ways to build Q: (1) using prototypes defined in Eq. (12) (K samples): (2) using all samples defined in Eq. (5) (10 * K samples): (3) randomly sampling one meta performs best, and the performance with random-sample meta or whole meta is still competitive, which demonstrates the robustness of our proposed method to the distribution Q and the benefit of using the prototypes to build Q. Third, we compare two ways for learning w in each itcration, where one is re-estimating w from scratch and another one is maintaining and updating w throughout the training (i.e., iteratively optimizing weights). We find that iteratively optimizing performs better.
cost can still achieve acceptable performance,
point from each class (K samples) in meta set. We find that prototype-
Since cost function is essential in optimizing the OT loss, we are interested in examining the learned weight vectors given by different cost functions. Here, we use CIFAR-LT-10, randomly choose {10, ..., 1} training samples from class {1, .... 10} and obtain 55 samples, which are used to build distribution P. Besides, the 10 prototypes from meta set are used to build the distribution Q. Given the different cost functions, we show the learned weight vectors of 55 training samples in Fig. I. which have very different properties. Specifically, the label-aware cost and feature-aware cost lead to class-level weights and sample-level weights, respectively. It is reasonable that label-aware cost
7

---

Table 1: Test top-1 errors (%) of ResNet-32 on CIFAR-LT-IO 10 and CIFAR-LT-I00 under different settings.
Datasets
CIEAR-LI-10
CHEAR-LT-100
Imbalance Factor
200
100
50
20
200
100
50
20
losstresults
29.86
17.56
61.54
55.98
Focal loss 12
(results from [
34.71
29.62
23.29
17.24
64.38
61.59
55.68
48.05
CB. CE loss
13
(results from [
31.23
27.32
21.87
15.44
64.44
61.23
55.21
48.06
CB, Focal
(results
25.43
60.40
54.79
LDAM loss 1
(results from
33.25
26.45
21.17
16.11
63.47
59.40
53.84
48.41
LDAM-DRW
17
fresults from 5
25.26
21.88
18.73
15.10
61.55
57.11
$2.03
47,01
[15 (resalts from 16]
33.49
21.07
59.77
55.56
Meta Meih-weight net [6
32.80
26.43
20,90
15.55
63.38
58.39
54.34
46,96
Meta-class-wee weight with CE Ioss
29.34
23.59
19.49
13.54
60.69
56.65
51.47
44.38
Meta-class-weigg with focal loss
25.57
17.12
55.30
44.27
Meta-class-weight with LDAM loss
[
22.77
20.00
17.77
15.63
60.47
55.92
50.84
47.62
AugaSAug with CE loss S
23.11
19.46
15.97
12.36
60.06
53.13
48.10
42.15
IB 08
27.85
23.47
18.34
14.59
60.34
54.61
51.07
46.43
IB+CB [X
30.04
24.03
17.91
14.73
60..1
54.73
51.20
46,58
Focal loss
25.88
22.03
17.62
14.32
59.61
55.04
51.08
45.47
MetaSAng with focal loss
22.73
19.36
13,96
12.84
59.78
54.11
48.38
42.41
Aut with LDAM loss [5]
22.65
19.34
15.66
11.90
56.91
51.99
47.73
42.47
BBN [25]
20.18
17.82
57.44
52.98
method (Weight Vector)
15.54
51.46
910
Feature Cost Matrix
Label-aware Cost Matrix
Combined
0.0
Learned Weight Vector
Leamned Weight Vector
Learned Weight Vector
Figure Learned weight vectors (bottom) given different cost functions (top) on CIFAR-LT- 10, where x-axis denotes the 55 samples from the current mini-batch and where we only mark their labels for clarity.
only decides whether the two samples (from the meta set and training set) belong the same class, resulting in class-level measure. However, feature- aware cost measures the distance between samples from the sample-leve,, where each sample has own feature. More interestingly, the learned weights with the combined cost own the characteristics of class-level and sample-level weights simultaneously, where example weights of different classes are far away and example weights of the same class are close. Coincidentally, using the combined cost to define the OT loss can reach the same goal of [4] which explicitly considers class-level and sample-level weight. Besides, we find that the learned example weights of the minority class are usually more prominent than those of the majority classes.
Table 3: Test top-1 errors (6) of ResNet- 152 on Places-LT.
Table 2: Ablation study CIFAR- LT-100 with 200, where w maintained and updated throughout the training except the last row.
Table 4: Test top-1 errors (%) of ResNet-50 on ImageNet-LT. in-
Places-LT
dicates results from [5].
CF
69.3
Focal loss [2
(from
24
65.4
Method
ImzgeNet
Combined + Prototype scrotch
Combined RRndoo sample
Combined Protatype
Peature Prototype
Combined+ Whole
label Protstype
Method
Top-1 errors
$4.97
55.06
55.04
55.07
55.03
Mers-class-epp weighl
1,RW RANDOM
Mata-softmox
DisAlign
Our method
LWS
cRT
6
C
F5
60.32:10.02
67:77
644.1
62.4
62.9
61.3
60.7
Our RorlheedRReddass Pretetype
LDAM DRW
Merud Aug. CE
CB. CE CB:
Our methou
$2.36:3001
$4.26
61.12
$5.66
$2.61
$9.15
$2.41
54
To verify whether our method ameliorates the performance on minority classes, we plot the confusion matrices of CE, MetaS Aug, and ours on CIFAR-LT-IO with IF= 200 in Fig. 2 As expected, although CE training can almost perfectly classify the samples in majority classes, it suffers severe performance degeneration the minority classes. MetaSAug improves the accuracies of the minority classes, where is still big gap between the performance the minority classes and the majority classes. In

---

contrast, ours does not show a very clear preference for certain class and outperforms the strong baseline on the overall performance, which is the goal of on an imbalanced classification task.
Experimental details and results on Places-I.T and ImageNet-ILT Following [6] we employ ResNet-152 pre-trained on the full ImageNet as the backhone Places-LT.. For stage we set the initial learning rate as 0.01, which is decayed by le-1 every 10 epochs. In the stage 2 of our method, we only fine-tune the last fully-connected layer for training efficiency and set a le- and B as within within 50 epochs. The mini-batch size is 32 and the optimizer is SGD with momentum 0.9 and weight decay 5e 3 As shown in Table 3 our method outperforms all baselines. It further suggests that our method has excellent performance in the extreme imbalance setting with 1F=4980/55 For a fair comparison, we implement our method on ImageNet-LT with the same experimental conditions of [3], from which we have taken the results of other comparison methods. We consider 5e 40 as the backhone on ImageNet-LT. In stage 1, we run 200 epochs and decay the learning rate by 0.1 at the 60th and 80th epochs. In stage 2, we implement our method for 50 epochs, set learning rate a as 20-5 and 3 as 1e-2, and only fine-tune the last fully-connected layer for training efficiency. We use the SGD optimizer with momentum 0.9, weight decay 5e and set the batch size as 128. The results on ImageNet-LT of different models reported in Table Lindicate the effectiveness of our proposed method on ImageNet-LT when comparing with strong baseline MetaSAug, Besides, we further consider randomly sampling a mini batch of size 100 from all prototypes at each iteration to build Q. whose performance is comparable to the Q from all prototypes. Thus, with a stochastic setting for Q, our proposed method can be used to the imbalanced training set with a large number of classes. We defer the time computational complexity, additional quantitative results and qualitative results on different image datasets to Appendix B
Cross
MetaSAug
Ours
oon om 29 D6 0.00 000 02
DuR 92 0.00 6: DO D1 0:00
8.550000 01
0000
6:05 0.
0U 0.00
28 00 0.00 0:00 02
80 0.00
0.00 6: C 0:00 0.3 0.03
0.00
0.000000 0.08 D 0.00 6:03
0.93
27
23 8: 0.1
040
04
0100
0100 6:01 0:03
65
0:00
0:00
OOL
000
0:3
0.02
0.03 0/02
0.03 0.32
True label
0.04 002 0.9 32 02 on2
10000000 9.92
0: 082 002
60 0000000 0.00
0.02 0.04
0:0 9.1
912 000000 11 9:3 00 1000
6:00 0.03 53 0.000000000000000000000000000000 c
3.00
03 0:03
0.02 03 6600000000000
557 000001
000001
02 ozE 22 co u au uronisee 01
R22 1. 55 0:1 oU 8 c0
0:00
R. 8:02 001 8 nat 0st
Bock
0.22
0.00
0000000
30 0.03 0.00 8:0 0.1
8.99 0.00001 8 0.11
Predicted label
Predicted label
Predicted label
Figure 2: Confusion matrices of the cross-entropy training, MetaS and ours on CIFAR-LT-IO with the imbalance factor 200. We rank classes by the frequency, i.e., frequent (left) and rare (right).
6.2 Experiments on Imbalanced Text Classification
Datasets and settings Following D 2], we adopt the popular SST-2 for 2-class and SST-5 for 5-class sentence sentiment 43). For fair comparison, we use the same imbalanced datasets and settings with 22. Specifically, we set class 1 as the minority class and the rest as the majority classes, where the number of examples in the majority class is fixed as 1000 (SST-2) and 500 (SST-5) and we achieve different imbalance settings by varying the number of examples in the minority class. Besides, the number of samples in the meta set is 10 for each class. We use the BERT (base, uncased) model 44 as feature extractor and a simple 3-layer fully-connected network (FCN) with the structure in Cas as classifier. To make subsequent experiments on strong models, following (2]. we use an additional balanced training set (500 samples in each class) to fine-tune the BERT model, which is randomly selected from the remaining examples in each dataset except the imbalanced training set, meta set and to-be evaluated test set. Based on the fine-tuned BERT, we adopt the two-stage manner for the imbalanced text datasets, where we train the BERT + FCN in the first stage with CE loss and train the FCN with our proposed method by freezing the BERT in the second stage. The settings of the training process are deferred to Appendix C
Appendix[C
Baselines We consider the following methods: (1) vanilla BERT, the vanilla pretrained language model. (2) Fine-tuned BERT, where the pretrained BERT fine-tuned on an additional balanced training set. (3) Fine-tuned BERT + CE, the fine-tuned BERT model followed by the FCN which
9

---

is further trained by the CE loss on the imbalanced training set following [ 2 (4) Automatic re-weighting methods, including the method of Hu et al. [T] and constraint-based re-weighting [ Since few works consider imbalanced text classification, we further consider (5) Empirical re- weighting methods, including re-weighting with inverse class frequency (i.e., Proportion) (1)
and L.DM-DRWW [17] and (6) Logit adjustment [20] using their official codes and settings We repeat all experiments 10 times and report the mean and standard deviation.
Experimental details and results on SST-2 and SST-5 We report the text classification results of compared methods under different imbalance factors in Table [ We find that our proposed method outperforms all competing methods in all imbalance factor settings, which demonstrates the effectiveness of our proposed method. Although all methods could achieve acceptable performance in slight imbalance. the performance of three baselines (Vanilla BERT, Tuned BERT and Fine- Tuned CC) drop dramatically, indicating the importance of proposing specialized methods for handling imbalanced training datasets. Logit adjustment (post-hoc comrection), is very competitive to ours on SST-2, which, however, only produces similar results to the three above-mentioned baselines on SST-5, In contrast, ours is robust to not only the imbalance factors but also the number of classes, where the results are consistent with the image case. We provide more results in Appendix C.4 In addition to ID text and 2D image, we further investigate the robustness of our method on 3D point cloud data, where we use the popular ModelNet10 [45] and defer the experiments to Appendix D
Table 5: Comparison of different models on SST-2 and SST-5, t indicates results reported in 2
Method Imbulance Factor
SST-2
SST.5
1000 100
1000: 50
1000: 20
1000 10
500 75
500 60
300:
Vanilla BERT
74.91=4..2
53.2611:.70
10.54=1.40
49.8420.02
36.99:0.46
36.75=0.43
36.4620046
Tine-Tunee BERT
8166423.79
75.5341.90
65.23.3.91
60.115..00
43.76.10.77
43.25.00.73
42.7010.52
Tine-Tuned RERT cCE
78.25+2.24
57.18+1.88 1.88
58.00+11.23
50.17+1.34
43.71-0.98
44.06-1.11
36.46+00.50
Proportion treported by us)
79.1511.34
76.84.11.3
73.611117
14.4
42.781082
42.36=1106
41.6011.3
LDAM-DRW [7 (reported by us)
71,41+1225
64.65++3.82
56.41+3.55
53.73+3.01
43.20-00.64
40.90-1.66 .66
4088+ 46
Hu et al''s [
81.57.0.74
79.35.12.59
73.611
$5.84111.8
39.82.11.07
all ReReellaaiizziiionn 0
82.25:1.116
79.5321.64
66.68114.0
40.1410.399
Constraint- bused ze- weighigtiing"
2
82.58-10.88
81.1411.25
80.62.10.33
44.62.11.08
Logit Adjustment [0 (reported by us}
86.3710.30
86.6110.11
86.3110.3
86.30:0033
43.5222633
39.322203
36.3512.77
Our method (Weight Net)
87.08=0.099
87.13410.04
87.14:0.08
87.10=:0:05
44.95:0.56
44.79=0.82
44.68=0.98
7 Conclusion
This paper introduces a novel automatic re-weighting method for imbalance classification based on optimal transport (OT). This method presents the imbalanced training set to-be-learned distribution over its training examples, each of which is associated with a probability weight. Similarly, our method views another balanced meta set as balanced distribution over the examples. By minimizing the OT distance between the two distributions in terms of the defined cost function, the learning of weight vector is formulated as a distribution approximation problem. Our proposed re-weighting method bypasses the commonly-used classification loss on the meta set and uses OT to learn the weights, disengaging the dependence of the weight learning on the concerned classifier at each iteration. This is an approach different from most of the existing re-weighting methods and may provide new thoughts for future work. Experimental results on a variety of imbalanced datasets of both images and texts validate the effectiveness and flexibility of our proposed method.
Acknowledgements. This work is partially supported by grant from the Shenzhen Science and Technology Program JJCY/2021032412000 1032) and Shenzhen Institute of Artificial Intelligence and Robotics for Society.
10

---

References
[1] Zhiting Hu, Bowen Tan, Russ R Salakhutdinov, Tom M Mitchell, and Eric P Xing. Learning data manipulation for augmentation and weighting. Advances in Neural Information Processing Systems, 32, 2019.
[2] Yuqi Liu, Bin Cao, and Jing Fan. Improving the accuracy of learning example weights for imbalance classification. In International Conference Learning Representations, 2022.
[3] Kemal Oksuz, Baris Can Cam, Sinan Kalkan, and Emre Akbas. Imbalance problems in object detection: review. IEEE Trans. Pattern Anal. Mach. Intell,, 43( 33883333388-31,, 2021. doi: 10.1109/TPMM..00 2020 2981 890. URL https:///oo..or//10.... htt.://bbo.oo/11111000//M......0 100.11/0////M..20002222.1.
[4] Muhammad Abdullah Jamal, Matthew Brown, Ming-Hsuan Yang. Liqiang Wang, and Boqing Gong. Rethinking class-balanced methods for long-tailed visual recognition from a domain adaptation perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7610-7619, 2020.
[5] Shuang Li, Kaixiong Gong, Chi Harold Liu, Yulin Wang, Feng Qiao, and Xinjing Cheng. Metasaug: Meta semantic augmentation for long-tailed visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2021.
[6] Bingyi Kang. Saining Xie, Mareus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long- tailed recognition. In 8th In- ternational Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://oppnneeview.. wwtp:///ooonnddrrrgrrre....
[7] Nitesh V. Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. SMOTE: synthetic minority over-sampling technique. J. Artif. Intell. Res., 16:321-357, 2002. doi: 10.1613/jair.953, URL https://dooii////// 10. 161 3/jair.953
[8] Hui Han, Wenyuan Wang, and Binghuan Mao. Borderline- A new over-sampling method imbalanced data sets learning. In De-Shuang Huang, Xiao-Ping (Steven) Zhang, and Guang-Bin Huang, editors, Advances in Intelligent Computing, International Conference on Intelligent Computing, ICIC 2005, Hefei, China, August 23-26, 2005, Proceedings, Part
I, volume 3644 of Lecture Notes in Computer Science, pages 878-887. Springer, 2005. doi:
10. 15007/1133805/000 91. URL http::///oi..og/11...... 1007/11110///1 91
[9] C. Drummond and R. C. Holte. C4.5, class imbalance, and cost sensitivity: Why under-sampling beats over-sampling, proc of the ieml workshop learning from imbalanced datasets if, 2003.
[10] R. Barandela, R. M. Valdovinos, JS Sánchez, and F. J. Ferri. The imbalanced training sample problem: Under or over sampling? In Structural, Syntactic, and Statistical Pattern Recognition, Joint IAPR International Workshops, SSPR 2004 and SPR 2004, Lisbon, Portugal, August 18-20,
2004 Proceedings, 2004.
[11] C. Huang, Y. Li, C. C. Loy, and X. Tang. Learning deep representation for imbalanced classification. Computer Vision Pattern Recognition, pages 5375-5384, 2016.
[12] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[13] Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9268 9277, 2019.
[14] Yu-Xiong Wang. Deva Ramanan, and Martial Hebert. Learning to model the tail. Advances in Neural Information Processing Systems, 30, 2017.
[15] Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International conference on machine learning, pages 4334 4343. PMLR, 2018
11

---

[16] Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta- weight-net: Learning an explicit mapping for sample weighting. Advances in neural information processing systems, 32, 2019.
[17] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-awaa margin loss. Advances neural information processing systems, 32, 2019.
[18) Seulki Park, Jongin Lim, Younghan Jeon, and Jin Young Choi. influence-balanced loss for imbalanced visual classification. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 715-724. IEEE, 2021.
doi: 10.1 (..110//CV44992...0 2021 100077. URL https:////// orr/ 10. 100/1///44499222000.00000
[19] Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for long- tailed visual recognition. Advances in Neural Information Processing Systems, 33:4175-4186, 2020.
[20]
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. arXiv preprint arXiv:2007.07314, 2020.
[21] Hanyu Peng, Mingming Sun, and Ping Li. Optimal transport for long- recognition with learnable cost matrix. In International Conference on Learning Representations, 2022.
[22] Gabriel Peyré and Marco Cuturi. Computational optimal transport. Found. Trends Mach. Learn., 11(5-6):355-60) 0075555-7,2250 2019.
[23] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International conference on machine learning, pages 2304-2311 PMLR, 2018
[24] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large- scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2537-2546, 2019.
[25] Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In Proceedings the IEEF/CVF conference on computer vision and pattern recognition, pages 9719-9728, 2020.
[26] Julien Reygner and Adrien Touboul. Reweighting samples under covariate shift using a wasserstein distance criterion. Electronic Journal of Statistics, 16(1):377833314 2022.
[27] Alain Rakotomamonjy, Rémi Flamary, Gilles Gasso, M El Alaya, Maxime Berar, and Nicolas Courty. Optimal transport for conditional domain matching and label shift. Machine Learning, 1156-1766) 2022.
[28] Rosanna Turrisi, Rémi Flamary, Alain Rakotomamonjy, and Massimiliano Pontil. Multi- source domain adaptation via weighted joint distributions optimal transport. arXiv preprint arXiv:2006. 1 12938, 2020.
[29] Tongtong Fang, Nan Lu, Gang Niu, and Masashi Sugiyama. Rethinking importance weighting for deep learning under distribution shift. Advances in Neural Information Processing Systems, 33:1 12007, 2020.
[30] Yuguang Yan, Mingkui Tan, Yanwu Xu. Jiezhang Cao, Michael K. Ng, Huaqing Min, and Qingyao Wu. Oversampling for imbalanced data via optimal transport. In The Thirty- -hird AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January February 1, 2019, pages 5605-5611 AAAI Press, 2019. doi: 10.1609/aaai, v33i01. 1331555530.56055 URL https://doi.org// 1609/aaai. htt99/aai.v33i00 v33i01.3 33301155305605
[31] Viet Huynh, Dinh Q Phung, and He Zhao. Optimal transport for deep generative models: State of the art and research challenges. In IJCAL, pages 4450-4457, 2021.
12

---

[32] He Zhao, Dinh Phung, Viet Huynh, Trung Le, and Wray Buntine. Neural topic model via optimal transport. In International Conference on Learning Representations, 2021.
[33] Dongsheng Wang, Dandan Guo, He Zhao, Huangjie Zheng, Korawat Tanwisuth, Bo Chen, and Mingyuan Zhou. Representing mixtures of word embeddings with mixtures of topic embeddings. In International Conference Learning Representations, 2022.
[34] Anh Tuan Bui, Trung Le, Quan Hung Tran, He Zhao, and Dinh Phung. A unified Wasserstein distributional robustness framework for adversarial training. In International Conference Learning Representations, 2022.
[35] Dandan Guo, Tian Long, He Zhao, Mingyuan Zhou, and Hongyuan Zha. Adap- tive distribution calibration for few-shot learning with hierarchical optimal transport. 2022.
https:/larii..rg/as//211.00000
[36] Dandan Guo, Long Tian, Minghe Zhang, Mingyuan Zhou, and Hongyuan Zha. Learning prototype-oriented set representations for meta-learning. In International Conference on Learn- ing Representations, 2022.
[37] Marco Cuturi. Sinkhor distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26:2292-2300, 2013.
[38] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[39] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages
Ieee, 2009.
[40] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
[41] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[42] Songyang Zhang, Zeming Li, Shipeng Yan, Xuming He, and Jian Sun. Distribution alignment: A unified framework for long-tail visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2361-2370, 2021.
[43] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642, 2013.
[44] Jacob Devlin, Ming- Weng-Wei&e Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 18/0. 04.04005. 2018.
[45] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912-1920, 2015.
[46] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6.1 When handling
imbalanced datasets of image classification, our proposed method inferior some competitive baseline method when the dataset is less imbalanced.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Appendix
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We ve read the guidelines and make sure submission adheres to the ethical standards.
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] Code will be submitted in the supplemental material.
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section O
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] In Scction 6.1 we report the average result of 5 random image experiments and omit standard deviation for the limited space, which is usually small (c.g., 1e-2). In Section (6.2. we repeat all text experiments 10 times and report the mean and standard deviation.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix B.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "erx4EObUCn7", "reviewer": "Reviewer_DJ7Q", "summary": "This paper argues that the major challenge for imbalanced classification is essentially the mismatch between the imbalanced training dataset and the balanced test set. ​Besides, the general optimizing the weights by the classification loss may lead to inaccurate learning of the weights. Therefore, the authors formulate the learning of weight vector as the distribution approximation problem by minimizing the OT distance between to-be-learned distribution of samples in imbalanced training set and another balanced meta set. To show the effectiveness of the proposed method, they conduct extensive experiments on imbalanced classification tasks including image, text, and point cloud. ", "questions": "1. ​Why does the dependence of weights on classifier at training stage may lead to inaccurate learning of the weights?\n2. Why are the weights obtained when the imbalanced training set is approximated to a balanced meta data better for training a classifier?\n3.  An analysis or discussion about the computation complexity of optimizing the OT distance is necessary.\n4. Missing the comparison with the highly related IB (Influence-balanced loss for imbalanced visual classification, ICCV 2021) and DisAlign (Distribution alignment: A unified framework for long-tail visual recognition, CVPR 2021.).\n5. Typos:\n1) $\\theta$ in Eq.(2) should be bolded.\n2) In Eq.(10), should $\\eta$-->$\\lambda$?", "limitations": "A hold-out balanced meta-data is required.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. The idea of sample-reweighting is not new. But the method proposed by the author that learning the sample weights by minimizing the OT distance between the imbalanced training set and a balanced meta set seems reasonably novel.\n2. The paper is well organized and easy to follow.\n\nCons:\n1. ​Why does the dependence of weights on classifier at training stage may lead to inaccurate learning of the weights? A further explanation is required to highlight the reasonability of the proposed method.\n2. The reasonability of the statement that \"the major challenge for imbalanced classification is essentially the mismatch between the imbalanced training dataset and the balanced test set\" also needs further consideration or explanation.\n3. An analysis or discussion about the computation complexity of optimizing the OT distance is necessary.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Tiw9Ou3Dnci", "reviewer": "Reviewer_LeMk", "summary": "In this paper, the authors proposed a reweighting method for class imbalance classification based on optimal transport. Experiments on both image and text data demonstrate the effectiveness of the proposed method. ", "questions": "1. I'm wondering if the method proposed in this paper can handle other settings, e.g., label noise as in the baseline L2RW?\n2. In Stage1 of the proposed method, if t1 is large, would it cause an overfitting issue by pre-training on an unweighted imbalanced dataset? How to decide the value of this parameter in practice? \n\n--------------Post-rebuttal comments--------------\n\nThank the authors very much for the rebuttal and the additional results!\n\nAfter carefully reading the rebuttal, some of my concerns are resolved by the additional experimental results of baselines. But my major concern still remains, i.e., the novelty of this paper may not be so strong given [1] as prior work. I agree with the authors that the two papers are the same in high-level thinking but different in the tasks. [1] considers a more general setting of distribution shift and this paper focuses on class imbalance. Also, the contribution of this paper includes using OT loss in class imbalance and empirical contributions on many datasets. I will slightly raise my score for this and the additional experimental results. \n\nHowever, the authors said the key difference is that this paper disengages the dependence of the weight learning on the concerned classifier while [1] does not. This may not be true. If I understand correctly, in [1] the weight learning is independent of the classifier when using hidden-layer-output transformation as the non-linear transformation of data. I would suggest the authors carefully check this point to avoid any potential misunderstanding about the position of this paper with related work.\n\n\n", "limitations": "It seems that the authors haven't adequately addressed the limitations and potential negative societal impact of the work.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is clear and well-written.\n2. The paper provides lots of empirical evidence to demonstrate the effectiveness of the proposed method.\n\nWeaknesses:\n1. This paper looks quite similar to an existing work [1]. \n- In [1], class imbalanced problem (i.e., called class-prior-shift in [1]) is studied as a case study under the same setting (i.e., also using a small balanced meta set), and according to importance weighting, the optimal weights can be computed directly (i.e., w = p_{meta}(y)/p_{train}(y)). I'm wondering why not use this simple method to compute weights if such a balanced meta set is available? \n- Also, compared with [1], the core difference may be that this paper learns weights by minimizing an OT distance while [1] is by minimizing a maximum mean discrepancy (MMD). What is the critical contribution of this paper?\n- In L2RW, their baseline RANDOM (i.e, using random weights from a rectified Gaussian distribution) is quite strong. It would be interesting to compare the proposed method with the RANDOM baseline. \n\n2. There is another crucial work worth to be referred and discussing in this area [2].\n\nReferences:\n\n[1] Fang et al. \"Rethinking importance weighting for deep learning under distribution shift.\" In NeurIPS 2020.  \n\n[2] Jiang et al. \"Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels.\" In ICML 2018.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HOGKbdWSIEI", "reviewer": "Reviewer_8X8s", "summary": "The authors propose a new sampling method to cope with imbalance learning, which is based on optimal transport. The advantages of the method are illustrated on several different types of datasets. Unlike the traditional reweight approach, the authors look at the problem from the perspective of distribution matching, which leverages the OT distance between the distributions to guide the learning of weight vector or weight net.", "questions": "See above.", "limitations": "See above.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "It is a very interesting idea to reweight the training samples based on OT in the training phase, this approach is therefore more flexible than the previous OT-based approaches to long-tailed problems. Whereas previous approaches have involved adjustments to the model's results in the post-processing stage, the authors go further and apply OT to the training stage and propose a corresponding solution. The authors' experiments are well developed and they validate the superiority of their approach on multiple data sets.\n\nIn addition to the above mentioned points, I have some concerns as follows:\n1) The technique presented by the authors for reweighting with OT actually proposes a similar approach in many domains. For example, the usage of OT to address domain adaptation under conditional domain matching and label shift is proposed in [1]. Similarly, reweighting of samples using OT is described in [1]. A similar approach is suggested in [2] and [3]. I hope the author can explain the difference between them.\n2) The authors do not seem to be comparing their experiments with the most advanced methods, and as far as I can tell, it seems that many recently published methods are better than the results in the authors' experiments. I wish the authors had compared their experiments with more methods than the ones in the paper.\n3) Also, it seems that the author's method can only be applied to classification. I wonder if the author's method can cope with imbalance in object detection, or object segmentation, and also if the method can be extended to imbalance regression? The authors should explain more about the limitations of this work.\n\nFrom the results on CIFAR and ImageNet, some of the authors' results seem to have very weak boosts, for example, almost less than 1%. The authors' method seems to be limited in terms of experimental effectiveness, and I would like to see the authors take multiple experiments and average them, and report the variance of the experiments. This is what the NeurIPS conference promotes the need to report. Of course, I note that the authors state that these values are reported for text classification, but I think the authors should report these values for all experiments.\n\nIn Table 2, the author has conducted comparative experiments on CIFAR-100 to compare which cost matrix is better, but some of the results do not appear to be very different from each other, so it is difficult to derive any significant differences between these different methods in terms of the results.\n\nThe writing of the paper also needs further improvement, some issues are as follows:\n1) The description of OT is very inadequate, which is very difficult to understand for many people from outside this field. I also hope that the authors will explain how Sinkhorn can be used to solve OT problems.\n2) The author should present Related Work after Introduction to make it easier for others to learn about related work in this field.\n3) There is some confusion in the author's notation format, e.g. for scalars, vectors, sets, parameters, etc. The author should have followed some notational conventions. It will make the text look more rigorous.\n\nIn line 313, \"... indicates the effectiveness\" should be \"indicate the effectiveness\".\n\n\n\n[1] Optimal Transport for Conditional Domain Matching and Label Shift.\n\n[2] Multi-source Domain Adaptation via Weighted Joint Distributions Optimal Transport.\n\n[3] Reweighting samples under covariate shift using a Wasserstein distance criterion.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Fr6p5MIzsQU", "reviewer": "Reviewer_4F4H", "summary": "Imbalanced classification is a significant problem that many real-world datasets are imbalanced and the model trained on these datasets can yield poor performance. To tackle this problem, this paper proposes a novel re-weighting method using optimal transport (OT). Using a balanced meta set, the weight vectors are learned by minimizing the distance between an imbalanced training set and balanced meta set. This paper demonstrates the effectiveness on multiple benchmark datasets including image and text classification.\n", "questions": "Please refer to the weaknesses.", "limitations": "The authors addressed the limitations and potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The idea of minimizing the distance between an imbalanced training set and a balanced meta set is intuitive, and easy to understand. \n- Re-weighting from optimal transport (OT) perspective is novel.\n- The proposed method shows good performance on multiple benchmark datasets, including image and text classification. \n- The paper is easy to follow.\n\nWeaknesses\n- The motivation of using OT for re-weighting is not fully convincing. What other automatic re-weighting methods have not been able to solve? And so, how OT can benefit ‘re-weighting’? \n- Using a balanced meta-set seems unfair compared with other methods which do not use a balanced meta-set. For example, ten training samples per class are selected from the CIFAR dataset. In the case of IF=100, the least frequent class has only 5 samples on CIFAR-100-LT. It means, the meta-set has more samples than the training set, which can give a significant information for the minority classes. For fair comparison, in my opinion, other methods should also additionally use the meta-set in their training. \n- This paper missed some important recent baselines. For example, [1] automatically re-weights samples using the gradients and does not require a meta-set. [2] proposed Balanced Softmax to fix the label distribution shift between training and testing. \n\n[1] Influence-Balanced Loss for Imbalanced Visual Classification, ICCV, 2021.\n\n[2] Balanced meta-softmax for long-tailed visual recognition, NeurIPS, 2020.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept