Optimal Transport for Long-Tailed Recognition with Learnable Cost Matrix
Authors: Hanyu Peng
Mingming Sun
Ping Li
Venue: ICLR 2022 Poster (2022)
Keywords: Long-tailed recognition
imbalanced classification
optimal transport

Abstract: It is attracting attention to the long-tailed recognition problem, a burning issue that has become very popular recently. Distinctive from conventional recognition is that it posits that the allocation of the training set is supremely distorted. Predictably, it will pose challenges to the generalisation behaviour of the model. Approaches to these challenges revolve into two groups: firstly, training-aware methods, with the aim of enhancing the generalisability of the model by exploiting its potential in the training period; and secondly, post-hoc correction, liberally coupled with training-aware methods, which is intended to refine the predictions to the extent possible in the post-processing stage, offering the advantages of simplicity and effectiveness. This paper introduces an alternative direction to do the post-hoc correction, which goes beyond the statistical methods. Mathematically, we approach this issue from the perspective of optimal transport (OT), yet, choosing the exact cost matrix when applying OT is challenging and requires expert knowledge of various tasks. To overcome this limitation, we propose to employ linear mapping to learn the cost matrix without necessary configurations adaptively. Testing our methods in practice, along with high efficiency and excellent performance, our method surpasses all previous methods and has the best performance to date.

Published as a conference paper at ICLR 2022
OPTIMAL TRANSPORT FOR LONG-TAILED RECOGNITION WITH LEARNABLE COST MATRIX
Hanyu Peng, Mingming Sun, Ping, Li
Cognitive Computing Lab Baidu Research
No.10 Xibeiwang East Road, Beijing 100193, China 10900 NE 8th St. Bellevue, Washington 98004, USA (penghanyu, s wunmingminginn http://mmmnnnn..iingggg.111
lipinglil (aapnnggg)aaaiiiiiaa @baidu......... com
ABSTRACT
It is attracting attention the long-tailed recognition problem, a burning issue that has become very popular recently. Distinctive from conventional recognition is that it posits that the allocation of the training set supremely distorted. Predictably, it will pose challenges to the generalisation behaviour of the model. Approaches to these challenges revolve into two groups: firstly, training-awarr methods, with the aim of enhancing the generalisability of the model by exploiting its potential in the training period; and secondly, post-hoe correction, liberally coupled with training- aware methods, which is intended to refine the predictions to the extent possible in the post- processocsssing stage, offering the advantages of simplicity and effectiveness. This paper introduces an alternative direction to do the post-hoc correction, which goes beyond the statistical methods. Mathematically, we approach this issue from the perspective of optimal transport (OT), yet, choosing the exact cost matrix when applying OT is challenging and requires expert knowledge of various tasks. To overcome this limitation, we propose to employ linear mapping to learn the cost matrix without necessary configurations adaptively. Testing our methods in practice, along with high efficiency and excellent performance, our method surpasses all previous methods and has the best performance to date.
INTRODUCTION
Classification problems in the real world are generally challenged by the long-tailed label distribution, i.e. having a small number of samples for majority of labels, and a dominant number of samples for a minority of labels (Van Horn & Perona, 2017; Buda et al., 201 8; Liu et al., 2019). It is also known imbalanced recognition, which has been widely studied in the past decades (Cardie & Nowe, 1997; Chawla et al., 2002; Qiao & Liu, 2009; Cui et al., 2019). These distribution biases pose significant challenge predictive modeling: conceivably, models often suffer from poor generalisation and undesirable estimation bias (Cao et al., 2019; Kang et al., 2020; Zhou et al., 2020).
Recently, renewed interest in the problem of long-tail recognition has emerged following the context of neural networks, as numerous publications in the literature endeavour to resolve the problem albeit in different ways including decouple (Kang al., 2020). meta-learnigg (Ren et al., 2020: Wang et al., 2020; Li et al., 2021), post-hoc correction (Tang et al., 2020; Hong et al., 2021), etc (Liu et al., 2019; Cao et al., 2019; Tang et al., 2020). One of the representative methods of post-hoc correction, Logit Adjustment Menon et al. (2021), provides statistical correction to the prediction, receiving widespread attention for its simplicity and validity. But the downside that it is conducted on individual samples, the rectified marginal distribution may not satisfy the desired distribution.
Figuring out exact flaws of Logit Adjustment, our explicit modeling of the problem mathematically tumss into equational constraint, meanwhile to minimise the difference between refined distribution and the original one, this minimisation is motivated upon the inner produut similarity. A little further, the resulting problem can be linked OT. Drawing on this linkage, we develop it further by proposing

---

Published as a conference paper at ICLR 2022
linear mapping automatically learn cost matrix, thereby cireumventing the requirement for expert knowledge to configure this matrix. In summary, our contributions are as follows:
We propose an alternative direction based on convex optimisation to do post-hoc correction, which goes beyond previous direction from the statistical view.
Imposing marginal distributions to align ideal ones, derive an optimisation problem tied to OT that is solved using Sinkhorn. More further, for better learning of the cost matrix, we present a linear mapping enabling elegant learning with one-layer network.
The experimental evidence shows the high efficiency and best performance on three bench- marks. It verifies that addressing the post-hoc problem via OT is helpful and effective.
PRELIMINARIES
In this section, we begin with notational definition, followed by an introduction to the long-taine tailed recognition problem. Finally, we briefly review the OT and Logit Adjustment Menon et al. (2021).
Notations: In what follows, for two matrices X,Y
RNNK we denote (X,Y) =
E Σ Ynk as the Frobenius dot-product. 8(-) stands for the Dirac function, p(-) repre- sents the probability distribution. Urr, c) [P E RNNK |P1x r, PT1y c}, where 1 and 1K are -idimension and K K-dimension vector whose elements are all 1. r and c refer to the vectors of size N and K, Urr, e) include all matrices with row and column sums and respectively.
2.1 PROBLEM FORMULATION
Having collection of training samples [<x ) validation samples { vh)) N. and test samples {(w,, ynn") '' for classification with labels and input x E Rdd long-tailed recognition assumes that the class-prior distribution for training data p(y²) is different from that for validation data p(y") and test data p(y'). Specifically, cong-tailed recognition means the distribution p(y") is highly skewed, that is, some classes have the dominant number of samples, while tailed labels own a very small number of samples. We can use imbalance ratio to measure the skewness in training data set, which can be defined as R Namss, Nma. where N N and N denote the largest and smallest number of samples in the training data set, respectively. In this paper, we assume that the marginal distribution of the test set is known, we consider it as an implicit prior knowledge to be applied. Stepping back, even if we not know the marginal distribution of the test dataset in advance. There are still ways to estimate the marginal distribution of the test dataset relatively precisely, such as methods in Hendrycks et al. (2018); Azizzadenesheli et al. (2019).
Obviously, most models trained on imbalanced training data set would suffer from extremely limited generalisation ability. Hence the ultimate goal is to learn a model that minimises the empirical risk:
N,
I ( (x (x;)),
C (((()))
N
where (xx)) E IR denotes logits with associated sample, (--) Rd + R represents the mapping via neural networks, L stands for the loss function, typically cross entropy for classification problem.
2.2 REMINDERS ON OPTIMAL TRANSPORT
OT used to calculate the cost of transporting one probability measure to another. We next present a brief introduction to OT to help us better view the long-tailed problem from an OT perspective.
For two random variables X and Y. we denote its corresponding probability measures as r and Besides, C(X,Y) X x Y + R4 stands for cost function which measures the expense of transporting X to Y. Based on these, we can define OT distance between X and Y as
d(r,c) min I X C(x, (xxdy *ell(ne))
(2)
2

---

Published as a conference paper at ICLR 2022
where (1, = [s (τ. y)dy = Jx x(2, y) = of is the joint probability measure with r and When extend the above to the discrete situation, we consider following discrete distributions:
E (o,)))((z))
Σ P: (y))((y))
where yı (zx;) and (()) represent the probability mass to the sample æ; and yj respectively. this context, OT distance can be expressed as:
dm (r,
min PEU(r.e)
(P.M).
(4)
where M stands for the cost matrix constructed by M.j = C( (,)) The goal of OT to find a transportation matrix P that minimizes the distance d c))
As we can see, OT is distance measure between two probability distributions under some cost matrix (Villani, 2008). However, when above optimisation problem, often comes at the cost of heavy computational demands. To tackle this issue, OT with entropy constraint is proposed to allow the optimisation at small computational cost in sufficient smoothness (Burges et al., 2013). By adding Lagrangian multiplier to the entropy constraint, the new formulation can be defined as follows:
network simplex interior point methods to solve the
(r, ( M) (P*,M)
where P min (P, M) Ah(P).
(5)
PEU((re)
where À E +00), (())
2==1 Pak log PAks dis (r, e) also known dual-Sinkhorn
divergence, besides, it can be calculated with matrix scaling algorithms for cheaper computational demand. The following lemma guarantees the convergence and uniqueness of the solution.
Lemma For X > O, the solution p> is unique and has the form p = diag( u1) K Kdiag (v), where and v are two non non-negative vectors uniquely defined to multiplicative factor and K = = M// the glement-wise exponential of MM/>.
The above lemma states the uniqueness of p> (Sinkhorn, 1974), and P> can be efficiently computed via Sinkhorn's 's fixed point iteration u, U t- rr/Kv, c./ KTu.
2.3 A QUICK RECAP OF LOGIT ADJUSTMENT
We give brief introduction Logit Adjustment (Menon et al., 2021; Hong al., 2021). For the model (--), it is trained by the standard cross- erosspyy loss function on imbalanced training data set, and evaluated on test data. this algorithm, the test logit is adjusted as follows:
(xx") (<x") logp(v")
This simple procedure is derived from the Bayes optimal rule. It is apparent that Logit Adjustment involves post hoc correction individual sample, which does not necessarily guarantee that the marginal distribution of the whole dataset matches the desired distribution.
3
METHODOLOGY
The first part of this section explores post-hoc correction from an OT perspective, proceeds to the automatic learning of the cost matrix via linear mapping. Lastly, demonstrate how it can be achieved simply with one-layer neural network.
3.1 POST-HOC CORRECTION FORMALISED FROM AN OT PERSPECTIVE
Since Logit Adjustment applies adjustment at the individual sample level. It doesn't assure that marginal distribution of the overall data set fulfils our desired distribution. In this respect, we clearly put the constraint into an equation:
Y μ.
3

---

Published as a conference paper at ICLR 2022
where Y E I NxK indicates the refined prediction value in matrix form, μ represents the expected distribution on the test set. Alternatively, it is desirable to preserve another characteristic of Y. namely, remaining almost similar to the original prediction possible. We consider inner- inner-product based similarity to measure this, which a straightforward yet useful similarity measure.
maximize (C(2). Y):
(8)
where 2 represents the original prediction in matrix form, C(-) denotes to some transformation to Z, it can be some simple function, like Logarithmic function log(z). exponential function z Here select - log(-) as the cost function. This choice was driven by the requirement that the cost matrix must be positive definite, whereas the transformation of the original prediction by - llg(()) satisfies this condition. In addition, as log likelihood represents the local probability density of the associated samples, it can also be used to substitute 2 for the similarity approximation. In brief, the resulting numerical form can be put in formal terms as follows:
minimize Y
(-1og(2),Y)
(9)
subject to Y - p4, Y1x 1x
(10)
Extra constraint on Y is imposed simply cos the tuned estimation has to fulfil the basic probabilistic requirement that its sum one. Comparing Eq. (9-10) with Eq. (4), we can see that if we substitute P with Y, and substitute and with 1n and fu respectively, find that the above optimisation problem is actually a special case of OT.
In preliminaries, the entropy regularised OT (EOT) is introduced. By adding entropy regularisation to OT, the given equation can be solved efficiently by Sinkhorn algorithm. Specifically, the equation is
minimiie
(-log(Z), (), Y) + AY (Y)
(11)
subject to YTIN p. Y1k 1n.
The associated algorithmic flow for solving Eq. (11) is outlined in detail Algorithm 1.
Algorithm 1: Solve OT-related algorithm efficiently in the post-hoc correction via Sinkhorn Algorithm.
Input: Cost matrix M log( 2), trade-off parameter X, max number of iterations Ny. iteration number t, error threshold e, current error a,, row and column sums r = 1y and C c= p4, |-| denotes the vector norm.
Result: Refined predictions Y
Initialise K = /M/ _Boldd = 1x, u = 1k, t = 0:
while t < N/ and 8 I do
u r./ rr/Ku: Ke;
C. KTu
a Uold u/:
Uold u:
t= t + 1;
end
Output Y diag (u) diaagu))diiag (diag(v)
Assigning À with I, it is observed that equate our objective function to the KL divergence, thus illustrating the extensive nature our approach. DARP (Kim et al., 2020a) has previously applied to long-tailed semi- sumi-supereieed classification.
We would like to illustrate the non-applicable scenarios of our method. Firstly, our method requires a large number of samples for evaluation. This is because if the batch size is small, we can not guarantee that the desired marginal distribution can be satisfied within the batch. In some online scenarios, the sample-wise correction method is more suitable. In addition, our method assumes that the marginal distribution is already known. We assume that is consistent with uniform distribution.
Remark
4

---

Published as a conference paper at ICLR 2022
3.2 COST FUNCTION LEARNING VIA LINEAR MAPPING
Simple functions are likely to be sub-optimal for the real data sets; this suggests the design of a better cost function to better fit the long-tailed recognition problem. However, manually designed cost functions require expert knowledge in different domains. Thus, we propose to use a linear mapping to automatically learn the cost function, which relieves the need configuration.
More specifically, for predictions Z generated by Softmax operation via leveraging linear transforma- tion matrix W, W E I x is learned so that the following objective function is minimised.
minimize YER
- -Y 2) + + log Y),
(12)
exp exp WT (æx,))s
Ylk= IN Y - n4 2n
Σ exp(WTo (In));;
The resulting formula can be illustrated using a simple one-layer network of weight parameter W, together with error function in Eq. (12). We initialise W with an identity matrix and use small learning rate to learn W. One could also absorb the term log p(y") as the fixed bias parameter into the network. Motivated by this description, we can use general gradient descent algorithm, such as SGD. to optimise the error function. Taking into account the implementation in practice, the direct calculation of Eq. (12) can be done using the Sinkhorn iterations (Burges et al., 2013; Frogner et al., 2015; Peyré et al., 2019) in mini-batch training efficiently. Besides, We term the proposed method as OTLM (Optimal T Transport Linear Mapping), Figure illustrates the overview of OTLM.
A single layer feed- forward network
Cost matrix -log(2)
Optimal Transport 14. 1
Refined prediction
Y
Input logits (xx)
Figure 1:
Our proposed framework OTLM. The logit 4 (x) is fed into single-layer cerw-forward network to infer Z. The cost matrix M is set to log Z. By solving the optimal transport problem via Sinkhom algorithm, we can obtain the refined prediction Y.
Compute the gradients w.r.t W With the optimisation in Algorithm 1, it conducted the overall data set with a comparatively large number of data. Quite different scenario now, as mini- batch training is more favoured when it comes neural networks. For this reason, the optimisation workflow in Algorithm 1, as pointed by Peyré et al. (2019); Viehmann (2019), will have problem of batch stablisation. To this end, we perform logarithmic transformation to step 3-4 in Algorithm 1.
log logrr log (M log logsumexp
M logg
(13)
1
log log log (M log c
logsumexp
M log
(14)
with log-sum-exp operation logsumexp (x)) log(>;; cxp 21). Assuming convergence is achieved in Sinkhorn's 's loops, it is unnecessary for us manually to compute the derivative of the loss function in Eq. (12) w.r/ W. As long as we make sure that the entire optimisation process is differentiable, modern deep learning libraries can automatically implement end-to-end derivations, as it allows us to differentiate between the results of Sinkhorn's 's loops mere composition of elementary operations.
Performance degradation caused by imbalanced distributions can be addressed with meta- meta-learning based methods (Ren et al., 2020; Wang et al., 2020; Li et al., 2021). Recent works have illustrated neu- ral networks can learn more meaningful representations from the validation data set {xx,, vh N. We also take a validation data set to optimise the parameter W, but with the still significant difference in that require labelling information, thus avoiding a large expanse of labeling samples.
5

---

Published as a conference paper at ICLR 2022
4 EXPERIMENTS
In this section, we first conduct experiments comparing our approach versus extant post-hoe correction methods on three data sets, including CIFAR- 10--LO (Cao et al., 2019), ImageNet-LT (Liu et al., 2019), and iNaturalist (Hor et al., 2018) with varying backbones. Finally, we empirically make comparison of our algorithm with alternative wutting-edge long-tailed recognition methods. Observe that, for one thing, a plausible coupling of our algorithm to any training-awar long-tailed recognition method exists. As an illustration of the potency and strong generalisation of our approach, we conducted post-hoc correction of the seeudo-predictions of both methods, which are fairly typical for the training stage of all data sets: The first is based the cross entropy (CE), and the other is the recently proposed RIDE (Wang et al., 2021). which is based on multi experts. We conducted experiments in the appendix for semi-supervised learning, some extent to imitate the online situation. All our experiments are implemented in the PaddlePaddle deep learning platform.
4.1 DATA SETS AND BASELINES
The baseline methods and data sets are briefly described here, with implementation details placed in appendix. We assume that the marginal distribution is uniform due the characteristics of data sets.
Baselines: We compare our methods with (i) post-hoc correction methods including Logit Adjust- ment (Menon et al., 2021), DARP (Kim et al., 2020a). (ii) state-of-the-ard methods including Focal Loss (Lin et al., 2017). LDAM (Cao et al., 2019), BBN (Zhou et al., 2020), Balanced Softmax (Ren al., 2020), Causal Norm (Tang al., 2020), LADE (Hong et al., 2021), M2m (Kim et al., 2020b), Decouple (Kang et al., 2020), LFME (Xiang et al., 2020), RIDE (Wang et al., 2021)
Long-tailed data set: We take experiments on three data including CIFAR-I00-LE ImageNet- LT, and INaturalist. We build the imbalanced version of CIFAR-100 100 by downsampling samples per class following the profile Liu et al. (2019); Kang et al. (2020) with imbalanced ratios 10, 50, and 100. For all the benchmarks, we evaluate the performance on the test data set using a model trained on the training data set, and report the results using top-1 1 accuracy.
4.2
MAIN RESULTS
Table illustrates the results on CIFAR-100-LT data set. For CE-based methods, it shows that Logit Adjustment is indeed a simple yet effective method for post-hoc correction. Remarkably, it outperforms the baseline methods by 2.6%, 4. 1., and 4.1% 1% under the imbalance ratio of 10, 50, and 100 respectively. With a quick look at the results based on the OT approach, the superior results stress the advantages of our approach over the DARP and Logit adjustments. OT algorithm can further improve the performance by 0.62%, 0.43% 0.70% for OTLM, it outperforms Logit Adjustment by 0.55%, 0.73%, 0.99% under the three imbalance ratios on CIFAR-100-LT data set. For RIDE-basee training-aware methods, there also 0.6% and 1.23% improvement in the accuracy of our method.
Table Comparison on the top-1 accuracy with post-hoc correction methods on CIFAR-100-LT data set using ResNet-32 backbone, ImageNet-LT and iNaturalist data set using (tteeeXXt-0-2x444 and ResNet-50 backbones respectively. Better results are marked in bold, the small red font indicates the increase in accuracy when our method is compared to Logit Adjustment and RIDE. '.' means the results are not reported. CE indicates that a cross-entropy loss function used in the training stage.
Data set
CIFAR- 100
ImageNet-LT
iNaturalist
Imabalced Ratio
10
50
100
Baseline (CE)
59,00
45.50
41.00
43.10
65.00
Logit Cdjustment+CF
61.63
49.62
45.11
51.62
69.44
DARP+CE
61.78
49.52
45.13
$1.20
71.18
OT+CE
62.25+0.62
50.05+0.43
4-.80
51.87
71,20+1.6
OTLM+CE
62. 68..5
50.35+0.73
46.10-099
52.35+a73
72.30+2.6
RIDE
50,20
57.50
72.90
OT+RIDE
50.82+062
58.45+095
75.85+295
OTLM+RIDE
51.43-1.23
58.71-121
76.12+3.22

---

Published as a conference paper at ICLR 2022
Table also provides the results ImageNet-LT and iNaturalist data sets. The fact again attracts our attention that for CE-based training-awer aware methods, OT, which is based on convex optimisation, consistently outperforms Logit Adjustment by large margin (1.76%) iNaturalist data set. The accuracy boosting indicates the great potential of methods based on optimisation for post-hoe correc- tion. Besides, not surprisingly, OTLM further enhance the prediction accuracy, it outperforms Logit Adjustment by 0.73% and 2.86% on ImageNet-LT and iNaturalist respectively. As for RIDE, the gain in accuracy is also significant, about 3% iNaturalist.
4.3 COMPARISON WITH THE STATE-OF-THEE THE- METHODS
Armed with demonstration of the validity of OT and OTLM, we switched to comparison of our performance with existing methods that achieved the most advanced results on three benchmarks. As shown in Table in particular, ImageNet-LT and iNaturalist data sets, the experimental results remarkable and impressive, our method outperforms RIDE by 1.0% and 3.0%, respectively. Since iNaturalist is very difficult and fine- traigga data consisting of 8, 8,142 categories. These huge performance gains come at a fraction of the cost, as we show in the following subsection.
Table 2: Comparison on the top-1 accuracy with hate-of-the-arte on CIFAR-100-LLT data set using ResNet-32 backbone, on ImageNet-LT and iNaturalist data using ResNeXt-50-32x44 and ResNet- 50 backbones respectively. denotes the results are not reported, results underlined are the ones being compared, best results are marked in bold, the small red font denotes performance gain.
Data Set
CIFAR- 100
ImageNet-LT
iNaturalist
Imabalced Ratio
10
50
100
Cross Entropy (CE)
55.7
45.5
38.3
44.4
61.7
Focal Loss
55.8
44.3
38.4
43.7
CB-Focal
61.1
OLTR
46.3
63.9
NCM
47.3
63.1
norm
49.4
69.3
CRT
49.6
67.6
LWS
49.9
69.5
LDAM
64.6
LDAM+DRW
58.7
46.6
42.0
68.0
BBN
59.1
47.0
42.6
69.3
Causal Norm
59.6
50.3
44.1
51.8
M2m
58.2
43.5
LFME
43.8
Balanced Softmax
61.6
49.9
45.1
LADE
61.7
50.5
45.4
51.9
70.0
RIDE (6 experts)
50.2
57.5
72.9
Logit Adjustment
61.6
49.6
45.1
51.6
69.4
DARP
61.8
49.5
45.1
51.2
71.2
OT+CE
62.3.a7
50.1
45.8
51.9
71.2
OTLM+CE
62.2
50.4
46.1
52.4
72.3
OT+RIDE (6 experts)
50.8
58.5.10
75.9
OTLM+RIDE (6 experts)
51.4.12
58.7-12
76.1.32
4.4 COMPUTATION COST OF OT AND OTLM
As we have highlighted, the additional computational cost of OT and OTLM is particularly small, compared to that of training-awan methods. In Table 3, exact time of the evaluations on the ImageNet- LT and iNaturalist data sets provided. Please note, the time here are measured from the start of each method to the final best performance. For the comparison between OT and OTLM, because OTLM runs on the GPU, one optimisation iteration of OTLM is in fact much faster than OT. Except for OTLM, which (2.20 Ghz Xeon). Firstly, we can observe that all post-hoc correction methods here are not quite time-consuming. For comparison, ImageNet-LT and iNaturalist can be trained on 4 NVidia cards
NVidia card (V 100), the results come from 28-core machine
7

---

Published as a conference paper at ICLR 2022
Table 3: Time for different methods exccute on the ImageNet-LT and iNaturalist data sets. The times here are measured from the start of each method to the final best performance. Their running time are counted in seconds. Coupled with the prior results, an observation can be made that OT can be the best coordinate to trade off performance and efficiency in post-hoc correction methods.
Data Set
ImageNet-LT
iNaturalist
Logit Adjustment
0.3
1.2
OT OTLM
32.7
82.5
134.4 256.5
(V100) for extremely long periods of time. Using ResNeXI-50-32x46 on the iNaturalist training data set, for example, with a batch size 256, the amount training time (in seconds) to perform iteration is approximately 850. Also, if one differs the performance of each method, gather that OT and OTLM provide the best balance of performance and running costs to match.
4.5 CONVERGENCE BEHAVIOR OF OTLM
To verify that our adopted OTLM algorithm is really efficient. With ResNeXt-50-32x4d the backbone, we plotted curves depicting the decreasing absolute error a with the number of iterations I on the ImageNet-LT validation data set. Image on the left Figure 2 gives us visualisation of the optimisation process. It is found that the absolute errrr converges at around 60 iterations. In fact, we discovered that 10 even fewer iterations were enough to yield satisfactory performance.
303
OTLM
102
Absolute Error
3812
10
100
10
2
10
0
10
20
30
40
50
60
lteration
Figure 2: The left image shows the convergence of OTLM:
The absolute error a decreases with iterations the ImageNet-LT validation data (20k samples), and used ResNet-50 as the backbone to obtain the original predictions Z. The right image demonstrates the block in W, we trained on CIFAR- 100 to produce the original predictions with OTLM to minimise the difference between the ideal and actual distributions. For ease of display, the values of the diagonal elements are relatively large and we will replace these values with zeros.
4.6 DETAILED ANALYSIS ON DISTRIBUTION MATCHING
Upon distribution fitting, preferably the estimation bias is lessened, which implies that the KL divergence distance between the refined prediction and the target distribution is decreased. Table displays the KI. distances on the ImageNet-LTT and iNaturalist data sets, and we can find that OT possesses a smaller KI. distance than Logit Adjustment, which suggests that OT decreases the estimation bias more than Logit Adjustment by much larger margin.
4.7 VISUALISATION ON WEIGHT PARAMETER IN OTLM
As natural consequence, one wonders about the exact structure of this weight matrix W, and no better way present its results than through visualisation. Towards this end, applied OTLM
8

---

Published as a conference paper at ICLR 2022
Table 4: KL distance between refined probability distribution and target distribution computed on ImageNet-LT and iNaturalist data sets. Smaller distance indicates two distributions are more similar.
Data Set
ImageNet-LT
iNaturalist
Logit Adjustment DARP OT
1.78e-05
5.81e-06
8.49e-08
1.07e-08
8.62c-10
4.72c-10
to refine the original predictions on CIFAR-100 data set with imbalanced ratio R 100 and ResNet-32 backbone, and trained a one-layer model until it converged to extract the value of W, where the shape of is 100 x 100. We visualised sub-block of size 20 20, the illustration is presented on the right image in Figure 2. The visualised matrix is very dense, this non-sparsity suggests that the OTLM manages correct the predictions of the model in the training stage, whilst minimising the distance between the desired and the actual distribution.
5 RELATED WORK
The great majority of current methods applied to long-tailed recognition can be divided into either training-aware and post-hoc correction methods. The training-aware methods aim to improve the generalisation ability of models trained on imbalanced training data set. Among them, re-sampling and re- weighting are two major methods. Re-sampling controls the class numbers by the means of over-sampling (Chawla et al., 2002) and under-sampling (Barandela et al., 2004; Drummond et al., 2003). The re-weighting approaches achieve balanced learning by giving smaller weights to majority classes and larger weights to minority classes (Tang et al., 2008; Zadrozny et al., 2003; Lin et al., 2017; Lee al., 2017: Cui et al., 2019; Khan et al., 2019). Other approaches have also been proposed to resolve the imbalanced problem. For example, Cao et al. introduces pevretcalll-princippe
Long-tailed recognition:
loss function motivated by minimizing the generalisation error bound (Cao et al., 2019), Kang et al. proposes decouple the learning phases into representation learning and classifier tuning (Kang et al., 2020). Apart from training-aware methods, post-hoc correction also offers avenue for long-tailed recognition problem, it be applied the normalisation of classifier weights (Kang et al., 2020; Zhang et al., 2019; Kim & Kim, 2020) or logits based on label frequency (Provost, 2000; Collell et al., 2016). Among them, Logit Adjustment (Menon al., 2021) is a simple yet powerful method to use. Compared with Logit Adjustment, our approach performs the post-hoc correction from the optimisation view.
Optimal transport: OT distance is important family of distances for probability measures (Villani, 2008; Santambrogio, 2015). It has been extensively applied in different fields in machine learning. Courty et al. (Courty et al., 2016) proposes a regularised unsupervised optimal transportation model to perform the domain adaption. Other applications include 3D shape matching (Su et al., 2015), generative model (Arjovsky et al., 2017; Salimans et al., 2018; Bunne et al., 2019; Deshpande et al., 2019; Bhagoji et al., 2019; An et al., 2021), graph matching (Xu et al., 2019a;b), model designs Kandasamy et al. (2018); Chizat & Bach (2018). To the best of our knowledge, we are the first to perform the post-hoc correction from the OT perspective.
6 CONCLUSION
In this paper, we present two techniques for post-hoc corrections from an optimisation perspective. Our approach reviews the basic mathematical formulation of the distribution alignment problem and relates the resulting formulation to OT, which we find, moreover, offers greater possibilities and flexibility to solve the problem. In order to avoid manual configuration of the cost matrix, propose to learn the cost matrix automatically by linear mapping, which leads to further performance improvements. The experimental results confirm the superiority of our proposed approach. We believe our work can open up more possibilities and imagination for post-hoc corrections. There are potential challenges that remain to be addressed. These include, for example, how to apply OT to correction of predictions under unknown marginal distribution. Meanwhile, an extension of our approach to other applications where label shift involved is considered possible.

---

Published as a conference paper at ICLR 2022
REFERENCES
Dongsheng An, Jianwen Xie, and Ping Li. Learning deep latent variable models by short-run MCMC inference with optimal transport correction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15415-15424, virtual, 2021.
Martín Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Proceedings of the International Conference on Machine Learning (ICML), pp. 214-223, Sydney, Australia, 2017.
Kamyar Azizzadenesheli, Anqi Liu, Fanny Yang, and Animashree Anandkumar. Regularized learning for domain adaptation under label shifts. In Proceedings of International Conference on Learning Representations (ICLR), New Orleans, LA. 2019.
Ricardo Barandela, Rosa Maria Valdovinos, José Salvador Sánchez, and Francesc J. Ferri. The imbalanced training sample problem: Under or over sampling? In Proceedings of the Joint IAPR International Workshops on Structural, Syntactic, and Statistical Puttern Recognition (SSPR and SPR), pp. 806, Lisbon, Portugal, 2004.
Arjun Nitin Bhagoji, Daniel Cullina, and Prateek Mittal. Lower bounds on adversarial robustness from optimal transport. In Advances in Neural Information Processing Systems (NeurIPS), pp- 7496-7508, Vancouver, BC, Canada, 2019.
Paula Branco, Luís Torgo, and Rita P. Ribeiro. A survey of predictive modeling on imbalanced domains. volume 49, pp. 1-50. ACM New York, NY, USA, 2016.
Mateusz Buda, Atsuto Maki, and Maciej A. Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, 106:249-259, 2018.
Charlotte Bunne, David Alvarez-Melin Andreas Krause, and Stefanie Jegelka. Learning generative models across incomparable spaces. In Proceedings of International Conference on Machine Learning (ICML). pp. 851-861, Long Beach, CA, 2019.
Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in neural information processing systems (NeurIPS), pp. 2292-2300, Lake Tahoe, Nevada, United States, 2013.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Aréchiga, and Tengyu Ma. Learning imbalanced datasets with labelldistribution-aaaar aware margin loss. In Advances in Neural Information Processing Systems (NeurIPS). pp. 1576-1576, Vancouver, Canada, 2019.
Claire Cardie and Nicholas Nowe. Improving minority class prediction using case-specific feature weights. In Proceedings of the International Conference on Machine Learning (ICML, pp. 57-65, Nashville, Tennessee, USA, 1997. Morgan Kaufmann.
Nitesh V Chawla, Kevin W Bowyer, Lawrence Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research (JMLR), 16:321-357, 2002.
Lénaïc Chizat and Francis R. Bach. On the global convergence of gradient descent for over- parameterized models using optimal transport. In Anmual Conference on Neural Information Processing Systems, (NeurIPS), pp. 3040-3050, Montréal, Canada, 2018
Guillem Collell, Drazen Prelec, and Kaustubh R. Patil. Reviving threshold simple plug-in bagging ensemble for binary and multiclass imbalanced data, arXiv preprint arXiv: 1eXiv:/606.000 (606008698. 2016.
Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamonjy. Optimal transport for domain adaptation. IEEE transactions on pattern analysis and machine intelligence (PAMI), 39(9): 1853-1865, 2016.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge J. Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9268-99687777 Long Beach, CA, USA, 2019.
10

---

Published as a conference paper at ICLR 2022
Ishan Deshpande, Yuan-Ting Hu, Ruoyu Sun, Ayis Pyrros, Nasir Siddiqui, Sanmi Koyejo, Zhizhen Zhao, David A. Forsyth, and Alexander G. Schwing. Max-sliced wasserstein distance and its use for gans. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10648-10656, Long Beach, CA, USA, 2019.
Chris Drummond, Robert C Holte, et al. C4. 5, class imbalance, and cost sensitivity: why under- sampling beats over-sampling. Workshop on learning from imbulanced datasets II volume 11, pp. 1-8. Citeseer, 2003.
Charlie Frogner, Chiyuan Zhang, Hossein Mobahi, Mauricio Araya-Polo, and Tomaso A. Poggio. Learning with a wasserstein loss. In Advances in neural information processing systems (NeurIPS), Montreal, Quebec, Canada, 2015.
pp.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 770-778, Las Vegas, NV, USA, 2016.
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks labels corrupted by severe noise. In Advances in neural information processing systems (NeurIPS), pp. 10477-10486, Montréal, Canada, 2018.
Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang. Disentangling label distribution for long-tailed visual recognition. Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition (CVPR), pp. 6626-6636, virtual, 2021.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge J. Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE Conference on Computer Vision and Paltern Recognition (CVPR), pp. 8769 8778, Salt Lake City, UT, USA, 2018
Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for imbalanced classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5375-5384. IEEE Computer Society, 2016.
Kirthevasan Kandasamy, Willie Neiswanger. Jeff Schneider, Barnabás Póczos, and Eric P. Xing. Neural architecture search with bayesian optimisation and optimal transport. In Anmual Conference on Neural Information Processing Systems, (NeurIPS), pp. 2020-2029, Montréal, Canada, 2018.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. 2020.
Salman Khan, Munawar Hayat, Syed Waqas Zamir, Jianbing Shen, and Ling Shao. Striking the right balance with uncertainty. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 103- 112, Long Beach, CA, USA, 2019.
Byungju Kim and Junmo Kim. Adjusting decision boundary for class imbalanced lcarning. 2020.
Jachyung Kim, Youngbum Hur. Sejun Park, Eunho Yang, Sung Ju Hwang. and Jinwoo Shin. Distri- bution aligning refinery of pseudo-label for imbalanced semi-supervised learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 1565-1576, Vancouver, Canada, 2020a.
Jachyung Kim, Jongheon Jeong, and Jinwoo Shin. M2m: Imbalanced classification via major- to to-minor translation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 13896-13905, 13005, Seattle, WA, USA, 2020b.
Wonji Lee, Chi-Hyuck Jun, and Jong-Seol Soog-Seok Lee. Instance categorization by support vector machines to adjust weights in adaboost for imbalanced data classification. Information Sciences, 381:92-103, 103, 2017.
Shuang Li, Kaixiong Gong, Chi Harold Liu, Yulin Wang, Feng Qiao, and Xinjing Cheng. Metasaug: Meta semantic augmentation for long-tailed tailed visual recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 5212-5221, virtual, 2021.
11

---

Published as a conference paper at ICLR 2022
Tsung- Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense ob- the
ject detection. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2999-3007, Venice,Italy, 2017.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X. Yu. Large- long-tailed recognition in open world. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2537-2546, Long Beach, CA, USA, 2019.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In Proceedings of International Conference on Learning Representations (ICLR), Virtual Event, Austria, 2021. OpenReview.net.
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends@) in Machine Learning, 1((-6))355-6677, 2019.
Foster Provost. Machine learning from imbalanced data sets 101. In Proceedings of the A000 workshop on imbalanced data sets, volume 68, pp. 1-3. AAAI Press, 2000.
Xingye Qiao and Yufeng Liu, Adaptive weighted learning for unbalanced multicategory classification. Biometrics, 65(1):159-168, 2009.
Jiawei Ren, Cunjun Yu, Shunan Sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Balanced meta-softmax for long-tailed visual recognition. Advances neural information processing systems (NeurIPS), virtual, 2020.
Tim Salimans, Han Zhang, Alec Radford, and Dimitris N. Metaxas. Improving gans using opti- mal transport. In Proceedings of International Conference on Learning Representations, ICLR, Vancouver, BC, Canada, 2018. OpenReview.net.
Filippo Santambrogio. Optimal transport for applied mathematicians. Birkätseer, NY, 55(58-63): 2015.
Richard Sinkhorn. Diagonal equivalence to matrices with prescribed row and column sums. ii Proceedings of the American Mathematical Society, 45(2):195-198, 1974.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang. Han Zhang, Colin Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In Advances in Neural Information Processing Systems (NeurIPS), virtual, 2020.
Zhengyu Su, Yalin Wang, Rui Shi, Wei Zeng, Jian Sun, Feng Luo, and Xianfeng Gu. Optimal mass transport for shape matching and comparison. IEEE transactions on pattern analysis and machine intelligence (PAMI). 37(11):2246 2209, 2015.
Kaihua Tang, Jianqiang Huang, and Hanwang Zhang, Long-tailed classification by keeping the good and removing the bad momentum causal effect. Advances Neurul Information Processing Systems (NeurIPS), virtual, 2020.
Yuchun Tang, Yan-Qing Zhang, Nitesh Chawla, and Sven Krasser. Svms modeling for highly imbalanced classification. IEEE Trans. Syst. Man Cybern. Part B, (91)2281--888 2008
Grant Van Horn and Pietro Perona. The devil the tails: Fine-grained classification the wild. 2017.
arXiv preprint
Thomas Viehmann. Implementation of batched sinkhorn iterations for entropy-rrgularizzed wasserstein loss. arXiv preprint arXiv:1907.0172: 2019.
Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008.
Renzhen Wang, Kaiqin Hu, Yanwen Zhu, Jun Shu, Qian Zhao, and Deyu Meng. Meta feature modulator for long-tailed recognition. arXiv preprint arXiv:2008.03426 2020.
12

---

Published as a conference paper at ICLR 2022
Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella X. Yu. Long- saigetailed recognition by routing diverse distribution-wwer -ware experts. In Proceedings of International Conference on Learning Representations (ICLR), Virtual, 2021.
Liuyu Xiang, Guiguang Ding, and Jungong Han. Learning from multiple experts: Self-paced knowledge distillation for long-tailed classification.. In Proceedings of European Conference on Computer Vision (ECCV), pp. 247-263, Glasgow,UK, 2020. Springer.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 6626-6636, Honolulu, HI, USA, 2021.
Hongteng Xu, Dixin Luo, and Lawrence Carin. Scalable gromov-wasserstin learning for graph partitioning and matching. In Advances Neural Information Processing Systems (NeurIPS), pp- 3046-3056, Vancouver, BC, Canada, 2019a.
Hongteng Xu, Dixin Luo, Hongyuan Zha, and Lawrence Carin. Gromov-wasserstenn learning for graph matching and node embedding. In Proceedings of International Conference on Machine Learning (ICML), pp. 6932-694) 6941, Long Beach, California, USA, 2019b. PMLR.
Bianca Zadrozny, John Langford, and Naoki Abe. Cost-sensitive learning by cost-proportionate example weighting. In Third IEEE international conference on data mining (ICDM), pp. 435-442, Melbourne, Florida, USA, 2003. IEEE
Junjie Zhang, Lingqiao Liu, Peng Wang, and Chunhua Shen. To balance not to balance: A simple- yet-effective approach for learning with long-tailed distributions. arXiv preprint arXiv:1912.04486. 2019.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: gelateral-brance network with cumulative learning for long-tailed visual recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9719-9728, Seattle, WA, USA, 2020.
13

---

Published as a conference paper at ICLR 2022
A OPTIMAL TRANSPORT ON LONG-TAILED SEMI-SUPERVISED CLASSIFICATION
Together, with the extension our approach, we adapt to semi-supervised long-tailed classifica- tion tasks. We bring our approach together with the advanced semi- semi-supervised learning methods FixMatch Sohn et (2020). Adhering to the configuration in DARP Kim al. (2020a), we also create synthetically long-tailed variants of ZIFAR-100 with imbalanced ratio R = 20. The sample size of the unlabeled data twice as large as the labelled data, with the same proportion of imbal- ance between them. To assess the performance, we report two popular metrics: balanced accuracy (bACC) Huang et al. (2016) and geometric mean scores (GM) Branco et al. (2016). Experimentally. with the exception of an OT align the marginal distribution of the unlabelled data, all settings consistent with DARP, and we set to 0.005. The results are reported in Table It can be seen that approach also achieves much better performance semi-supervised learning tasks. It can somehow mimic the online scenarios. And the advantages of OT coping with such cases can be demonstrated.
Table 5: Comparison of classification performance (bACC/GM) CIFAR-100 with imbalanced ratio 20, We report standard deviation and mean values for each evaluation metric. Better results are marked in bold.
Method
CIFAR- 100
bACC
GM
DARP
54.940.05
46.440.41
OT
56.6+0.08
48.4
B PARAMETER STUDY OF À
We conduct an additional experiment on iNaturalist data set to study the effect of À on the final performance. The value of À is selected from {0.01, 0.1, 1.0.2.0, 5.03, all other experimental settings are the same, including various hyperparameters, network structure, etc. As illustrated in Figure 3, we can find that OT achieves the best performance when
75.0
73.5
72.0
70.5
1.0
A
Figure Visualisation of the accuracy change curve with À iNaturalist data
C IMPLEMENTATION DETAILS
The specific implementation details for each data set under the different methods are described below.
C.1 IMPLEMENTATION DETAILS OF CROSS ENTROPY
CIFAR-100-LT:
During the training stage, following the recipe of (Tang al., 2020; Hong et al., 2021), we apply SGD with batch size 256 and weight decay 0.0005 to train a ResNet-32 (He et al., 2016) model for 200 epochs, employ the linear warm-up learning rate schedule for the first five
14

---

Published as a conference paper at ICLR 2022
epochs. We also set the base learning rate to 0.2 and reduce it at epoch 120 and 160 by factor of 100. During the post-hoe stage, we use Adam with batch size 10k and weight decay 0.001 to train the one-layer feed-forwan network for 20 iterations. We set to 0.1, 200, A to 0.001. The learning rate is constant and set to 0.001.
ImageNet-LTT:
During the training stage, following the implementation of (Hong al., 2021), train the BesNeXt-50-32xee (Xie et al., 2021) for 90 epochs and perform a cosine learning rate scheme with an initial learning rate of 0.05. SGD is also employed optimise the neural network with batch size 256, weight decay 0.0005, and momentum 0.9. During the post-hoc stage, we use SGD with batch size 10k, weight decay 0.001 train the one-layer geed-forward network for 100 iterations. The constant learning rate is 0.2. We set À to 1.2, T to 7, A to 0.001.
During the training stage, ResNet-50 (He al., 2016) is chosen the backbone network. We use SGD with momentum 0.9, batch size 256 to train the network. We utilise the cosine learning rate schedule gradually decaying from 0.1 to During the post-hoc stage, we use SGD with batch size 23,826, weight decay 0.001, À to 1.0, A to 0.001 and 7 to 10. The constant learning rate is set to 0.05. As iNaturalist has no test data set, we directly report the performance on validation data set.
INaturalist:
C.2 IMPLEMENTATION DETAILS OF RIDE
The training details use in RIDE share the same the original paper, there 6 experts in RIDE on all three data sets.
CIFAR-100-LT We experimented with R 100 and ResNet-32 as the base network. SGD is utilised our solver with momentum 0.9, batch size 128, and epoch 200. The learning rate is initialised as 0.1 and decayed by 0.01 at epoch 120 and 160 respectively.
ImageNet-LT We take experiments with ResNeXt-50 backbone with 100 epochs. We set the initial learning rate to 0.1 and reduce by 10 at epoch 60 and 80. SGD with batch size 256 and momentum 0.9 is also adopted as the optimisation solver here.
iNaturalist We conduct experiments with ResNet-50 as backbone. The training details are the same as ImageNet-LT except from batch size 512.
15


Reviews:
[{"review_id": "UheZ49xbNJO", "reviewer": "Reviewer_vjwX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "General comments:\nThe overall idea proposed by the paper is interesting and seems well-motivated. Though applying optimal transport to deal with imbalanced data is not particularly new[1], the proposed method in a post hoc manner is generally more flexible. With extensive experiments, the method is demonstrated to be effective. \n\nStrengths:\n1. The paper is self-contained and the main idea can be easily grasped. The introduction clearly states the research question and the motivation of the work.\n2. Formalizing the post hoc correction from an OT perspective seems interesting and well-justified. How the authors draw the connection between OT and logit adjustment is particularly appealing to me. \n\nHowever, some substantial concerns should be addressed before the paper can be accepted.\n\nWeaknesses or Concerns:\n1. Does the paper inherently assume that the batch size during evaluation is rather large? I am asking this because the constraint in equation (5) may fail if there are only a few images (e.g. 2 images) in a batch. In practice, it is unlikely that we can always have a batch with the same distribution as the desired one. Sample-wise correction may have its downsides, but it fits in most setups.\n2. The training of OTLM seems to depend on the convergence of Sinkhorn’s algorithm. I am wondering how long would it take to train the OTML? For a post hoc approach, we generally expect it to be flexible. If the post hoc method involves learning from the data, I am not sure whether OTLM can generalize from one dataset to another. \n3. It seems to me that the variant, OTLM, is not well-justified. From Table 2 and Table 3, I notice that the performance of OTLM is not much different from OT(sometimes even worse) but with a significantly slower inference speed. I would be thankful if the authors could elaborate more on why do we need linear mapping and what has it learned.\n\nMinor Issues:\n1. Why the evaluation of OTLM is much slower than OT even on GPU? Isn’t there only one layer of feed-forward network?\n2. ‘our method outperforms by RIDE by 1.0% and 3.0%’ -> sounds a bit over-claimed to me since the results obtained by coupling RIDE.\n3. For iNaturalist, the training and evaluation of OTLM are both done on the validation set. It seems like an unfair comparison to me.\n\n[1] Yan et al., Oversampling for Imbalanced Data via Optimal Transport, AAAI 2019", "summary_of_the_paper": "This paper contributes an extension of post hoc correction of long-tailed recognition with Optimal Transport (OT). Unlike the previous work (e.g. logit adjustment) which focuses on sample-wise correction, this work, on the other hand, considers the marginal distribution of the overall data for correction. The method is further extended by learning a cost matrix. Their experiments show the effectiveness of optimal transport in the long-tail recognition problem via comprehensive comparison with previous works in terms of performance and efficiency.", "summary_of_the_review": "Despite some drawbacks on the assumption made and experiment evaluations, the paper would still benefit the community. I therefore lean towards a positive direction at this moment.\n\nUpdate:  As mentioned by reviewer uP2E, the strong assumption of knowing the marginal distribution of labels is not desirable in real-world scenarios. The proposed approach can be only applied to offline applications. I would recommend the authors include the following items in the future version:\n\n1. The performance of the proposed approach with an estimated marginal distribution;\n2. How can one apply the proposed method to online scenarios? (e.g. store evaluated samples to form a large batch)\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SZxsJIhdVBZ", "reviewer": "Reviewer_uP2E", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n1. Post-hoc adjusting the prediction via OT is novel.\n2. The proposed method is technically sound. By accurately aligning the predicted label distribution with the balanced label distribution (Table 4), the paper successfully outperforms the state-of-the-art post-hoc adjustment methods in terms of recognition accuracy (Table 1, 2).\n3. A convergence analysis for the proposed OTLM is provided (Sect 4.5), making the paper easier to be reimplemented and reproduced.\n\nWeakness:\n\n1. The described post-hoc correction may not be feasible in real-world applications. \n\n   In Sect 3.1, the authors assume that the \"expected distribution on the test set\" is known, followed by an explicit constraint $Y^T1_N=c$ for the optimization. In another word, the paper assumes the test set to have a balanced label distribution and optimize the predictions towards the balanced label distribution.\n\n   However, the assumption is not always true. The test set in many long-tailed datasets is not balanced, e.g., the LVIS dataset. Those datasets use balanced metrics, e.g., mAP, to measure the model performance instead. In this case, assuming that the test set's label distribution is known might be improper. \n\n   A follow-up question would be how to employ the method in real-world applications, where the test set could be small? If we try to make a single-image prediction, it may not seem possible to conduct the proposed OT optimization on only one data point. How should the method adapt to these scenarios?\n\n   Note also that the proposed method takes significantly longer inference time than compared methods (Table 3).\n\n2. Some empirical analysis is not sufficient. 1) In both Table 1 and Table 2, OTLM+RIDE seem to be missing. Can OTLM combine with ensemble-based methods? 2) What is the effect of the entropy regularization coefficient $\\lambda$ in Equation 9? There lacks of an ablation analysis.\n\n3. Some minor errors: 1) \"...how it can be achieved simply one-layer neural network.\" -> \"...how it can be achieved simply by a one-layer neural network.\" 2) \"As a matter of fact, as long as we make sure the entire optimization process is differentiable.\" The sentence seems to be unfinished. 3) \"DAPR\" in both Table 1 and Table 2 should be \"DARP\". 4) Table 1 and Table 2 substantially overlap with each other. They have inconsistent precisions as well. Merging them into one table could make it clearer. 5) Figure 1 is a little misleading. Should the feedforward network have the same amount of input and output (four in the case)?\n\n\n\n", "summary_of_the_paper": "This paper proposes to solve the long-tailed recognition by aligning the predicted label distribution with the test label distribution via Optimal Transport (OT).", "summary_of_the_review": "I recommend a borderline reject. Although using OT is a novel idea, my major concern is that it could be infeasible to employ the OT-based method in practice. Thus, I am unsure if the new direction is worth further exploration. \n\n=========== after reading author's latest revisions======\n\nI find the limitations of the methods adequately addressed in the latest revision. I agree with the authors that this paper's setting is consistent with some existing literature and the proposed method can be useful in some offline applications after reading the author's feedback. I would like to raise my recommendation from borderline reject to borderline accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1UdSwL83eGf", "reviewer": "Reviewer_fcxC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. The idea to perform post-hoc correction from the perspective of optimal transport seems novel. Even though the detailed operation in practice is not complicated. Such an idea is well supported by the existing theory guarantees. \n\n2. The results in the experiments also seem good. The proposed method can be applied to different baselines to boost their performance. \n\nWeakness\n1. Since this paper has many mathematical expressions, I suggest the authors give more detailed descriptions of the notations used. For example, in Section 2, what is r, c in U(r, c)? Is the c in equation (5) the same as c in U(r, c)?\n\n2. For definition 1 and 2, the author may remark more for better understanding, especially when the notions used are not directly related to the proposed method. The introduction of some notations also seems missing or hard to get. \n\n3. In equation (11), are the parameters W and Y optimized iteratively? The introduction of W also seems abruptly. A more natural and smooth transition between 3.1 and 3.2 may be needed. \n\n4. To be honest, according to the description in the introduction, I did not get well about the difference between the proposed method and the previous Logit Adjustment. Maybe more details are needed. \n", "summary_of_the_paper": "This paper proposes a new method for post-hoc correction in long-tailed recognition. Specifically, they leverage the idea of optimal transport (OT) and propose a linear mapping to replace the original exact cost matrix in OT problem. From the experiments, the proposed method can be combined with existing methods and boost their performance further. ", "summary_of_the_review": "Overall, the idea from OT to deal with long-tailed problems is novel. However, the writing of this paper makes it not very easy to understand or follow. Especially for people who don't have much background on OT. I thus give a score of 5.\n\n================================update===================================\n\nThe author addresses most of my concerns. Also, the latest version has admitted its limitations and put a more clear clarification on its advantages. I thus decide to raise the score to boardline accept.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agreed that the idea proposed by the paper is interesting and is well-motivated for handling long-tailed recognition problems. 
As suggested by the reviewers, it seems important that the limitations the paper be addressed in the final version of the paper.
Decision: Accept (Poster)