Year: 2021

Abstract:
It is attracting attention to the long-tailed recognition problem, a burning issue that has become very popular recently. Distinctive from conventional recognition is that it posits that the allocation of the training set is supremely distorted. Predictably, it will pose challenges to the generalisation behaviour of the model. Approaches to these challenges revolve into two groups: firstly, training-aware methods, with the aim of enhancing the generalisability of the model by exploiting its potential in the training period; and secondly, post-hoc correction, liberally coupled with trainingaware methods, which is intended to refine the predictions to the extent possible in the post-processing stage, offering the advantages of simplicity and effectiveness. This paper introduces an alternative direction to do the post-hoc correction, which goes beyond the statistical methods. Mathematically, we approach this issue from the perspective of optimal transport (OT), yet, choosing the exact cost matrix when applying OT is challenging and requires expert knowledge of various tasks. To overcome this limitation, we propose to employ linear mapping to learn the cost matrix without necessary configurations adaptively. Testing our methods in practice, along with high efficiency and excellent performance, our method surpasses all previous methods and has the best performance to date.

heading: 1 INTRODUCTION
text: Classification problems in the real world are generally challenged by the long-tailed label distribution, i.e., having a small number of samples for a majority of labels, and a dominant number of samples for a minority of labels (Van Horn & Perona, 2017; Buda et al., 2018; Liu et al., 2019) on the training data set. It is also known as imbalanced recognition, which has been widely studied in the past decades (Cardie & Howe, 1997; Chawla et al., 2002; Qiao & Liu, 2009; Cui et al., 2019). These distribution biases pose a significant challenge to predictive modeling; conceivably, models often suffer from poor generalisation and undesirable estimation bias (Cao et al., 2019; Kang et al., 2019; Zhou et al., 2020).
Recently, a renewed interest in the problem of long-tail recognition has emerged following the context of neural networks, as numerous publications in the literature endeavour to resolve the problem albeit in different ways including decouple (Kang et al., 2019), meta-learning (Ren et al., 2020; Wang et al., 2020a; Li et al., 2021), post-hoc correction (Tang et al., 2020; Hong et al., 2020), etc (Liu et al., 2019; Cao et al., 2019; Tang et al., 2020). One of the representative methods of post-hoc correction, Logit Adjustment Menon et al. (2020), provides a statistical correction to the prediction, receiving widespread attention for its simplicity and validity, but the downside is that its optimisation is conducted on individual samples, without any indication that rectified marginal distribution satisfies the desired distribution we are striving for.
Figuring out how to solve the above problem, our explicit modeling of the problem mathematically turns into an equational constraint, meanwhile to minimise the difference between refined distribution and the original one, this minimisation is motivated upon the inner-product similarity. A little further, the resulting problem can be linked to OT. Drawing on this linkage, we develop it further by proposing a linear mapping to automatically learn cost matrix, thereby circumventing the requirement for expert knowledge to configure this matrix. In short, we contribute as follows:
• We propose an alternative direction based on convex optimisation to do post-hoc correction, which goes beyond previous direction from the statistical view.
• Imposing marginal distributions to align ideal marginal distributions, we derive an optimisation problem tied to the OT that is solved using Sinkhorn, a highly efficient algorithm. More further, for better learning of the cost matrix, we present a linear mapping enabling very elegant learning with one-layer neural network.
• The experimental evidence shows the high efficiency of the algorithm, and the best performance achieved on three benchmarks, also supports our claim that addressing the post-hoc problem via OT is helpful and effective.

heading: 2 PRELIMINARIES
text: In this section, we begin by giving a notational definition, followed by an introduction to the longtailed recognition problem, and finally, we briefly review the OT, and give a quick recap of Logit Adjustment Menon et al. (2020). Notations: In what follows, for two matrices X,Y ∈ RN×K , we denote 〈X,Y 〉 =∑N n=1 ∑K k=1XnkYnk as the Frobenius dot-product. δ(·) stands for the Dirac function, p(·) represents the probability distribution. U(r, c) = {P ∈ RN×K+ |P1K = r,P ᵀ1N = c}, where 1N and 1K are N -dimension and K-dimension vector whose elements are all 1. r and c refer to the vectors of size N and K respectively, U(r, c) include all matrices with row and column sums r and c respectively

heading: 2.1 PROBLEM FORMULATION
text: Having a collection of training samples {(xsn, ysn)} Ns n=1, validation samples {(xvn, yvn)} Nv n=1 and test samples {(xtn, yn)t} Nt n=1 for classification with K labels and input x ∈ Rd, long-tailed recognition assumes that the class-prior distribution for training data p(ys) is different from that for validation data p(yv) and test data p(yt). Specifically, long-tailed recognition means the distribution p(ys) is highly skewed, some classes have the dominant number of samples, while tailed labels own a very small number of samples. We can use imbalance ratio to measure the skewness in training data set, which can be defined as R = N s max
Nsmin , where Nsmax and N s min denote the largest and smallest
number of samples in the training data set, respectively. In this paper, we assume that the marginal distribution of the test set is known, we consider it as an implicit prior knowledge to be applied. Stepping back, even if we do not know the marginal distribution of the test dataset in advance. There are still ways to estimate the marginal distribution of the test dataset relatively precisely, such as methods in Hendrycks et al. (2018); Azizzadenesheli et al. (2019).
Obviously, most models trained on imbalanced training data set would suffer from extremely limited generalisation ability. Hence the ultimate goal is to learn a model that minimises the empirical risk, which can be formulated as:
J (Φ (xsn) , ysn) = 1
Ns Ns∑ n=1 L (Φ(xsn), ysn) , (1)
where Φ(xsn) ∈ RK denotes logits with associated sample, Φ(·) : Rd → RK represents the mapping via neural networks, L stands for the loss function, typically cross entropy loss function for classification problem.

heading: 2.2 REMINDERS ON OPTIMAL TRANSPORT
text: OT is used to calculate the cost of transporting one probability measure to another. We next present a brief introduction to OT with the aim of helping us better view the long-tailed problem from an OT perspective.
Consider probability measures r and c of the two random variables X and Y respectively, when the cost function C(X,Y ) : X × Y → R+ is defined to denote the cost of transporting X to Y , the OT distance can be defined as :
d(r, c) = min π∈Π(r,c) ∫ X×Y C(x,y)π(x,y)dxdy, (2)
where Π (r, c) = {∫ Y π(x,y)dy = r, ∫ X π(x,y)dx = c } is the joint probability measure with r and c. When we extend the above continuous case to the discrete situation, we consider following discrete distributions:
r = N∑ i=1 pi(xi)δ(xi) c = K∑ j=1 pi(yj)δ(yj) (3)
where pi(xi) and pi(yj) represent the probability mass to the sample xi and yj respectively. In this context, OT distance can be expressed as:
dM (r, c) = min P∈U(r,c)
〈P ,M〉. (4)
where M stands for the cost matrix constructed by Mij = C(xi,yj). The goal of OT is to find a transportation matrix P that minimizes the distance dM (r, c)
As we can see, OT is a distance measure between two probability distributions under some cost matrix (Villani, 2008). However, when we use network simplex or interior point methods to solve the above optimization problem, it often comes at the cost of heavy computational demands. To tackle this issue, OT with entropy constraint is proposed to allow the optimisation at small computational cost in sufficient smoothness (Cuturi, 2013). By adding a Lagrangian multiplier to the entropy constraint, the new formulation can be defined as follows:
dλM (r, c) = 〈P λ,M〉 where P λ = arg min P∈U(r,c) 〈P ,M〉 − λh(P ), (5)
where λ ∈ [0,+∞], h(P ) = − ∑N n=1 ∑K k=1Pnk logPnk, d λ M (r, c) is also known as dual-Sinkhorn divergence, besides, it can be calculated with matrix scaling algorithms for cheaper computational demand. The following lemma guarantees the convergence and uniqueness of the solution of the algorithm.
Lemma 1 For λ > 0, the solution P λ is unique and has the form P λ = diag(u)Kdiag(v), where u and v are two non-negative vectors uniquely defined up to a multiplicative factor andK = e−M/λ is the element-wise exponential of −M/λ.
The above lemma states the uniqueness of P λ (Sinkhorn, 1974), and P λ can be efficiently computed via Sinkhorn’s fixed point iteration u,v ← r./Kv, c./Kᵀu.

heading: 2.3 A QUICK RECAP OF LOGIT ADJUSTMENT
text: We give a brief introduction to Logit Adjustment (Menon et al., 2020; Hong et al., 2020). For the model Φ(·), it is trained by the standard cross-entropy loss function on imbalanced training data set, and evaluated on test data. In this algorithm, the test logit is adjusted as follows:
Φ(xtn) = Φ(x t n)− log p(ys) (6)
This simple procedure is derived from the Bayes optimal rule. It is apparent that Logit Adjustment involves a post hoc correction on an individual sample, which does not necessarily guarantee that the marginal distribution of the whole dataset matches the desired distribution.

heading: 3 METHODOLOGY
text: The first part of this section explores post-hoc correction from an OT perspective, proceeds to the automatic learning of the cost matrix via linear mapping. Lastly, we demonstrate how it can be achieved simply with one-layer neural network.

heading: 3.1 POST-HOC CORRECTION FORMALISED FROM AN OT PERSPECTIVE
text: Since Logit Adjustment applies adjustment at the individual sample level. It doesn’t assure that the marginal distribution of the overall data set fulfils our desired distribution. In this respect, we clearly put the constraint into an equation:
Y ᵀ1N = µ, (7)
where Y ∈ RN×K indicates the refined prediction value in matrix form, µ represents the expected distribution on the test set. Alternatively, it is desirable to preserve another characteristic of Y , namely, remaining almost as similar to the original prediction as possible. We consider inner-product based similarity to measure the similarity, which is a straightforward yet useful similarity measure. In mathematical terms, this can be expressed as follows:
maximize Y
〈C(Ẑ),Y 〉, (8)
where Ẑ represents the original prediction in matrix form, C(·) denotes to perform some transformation to Ẑ, it can be some simple function, like Logarithmic function log(z), exponential function zα. Here we select − log(·) as the cost function. This choice was driven by the requirement that the cost matrix must be positive definite, whereas the transformation of the original prediction by − log(·) satisfies this condition, in addition, as log likelihood represents the local probability density of the associated samples, it can also be used to substitute Ẑ for the similarity approximation. In brief, the resulting numerical form can be put in formal terms as follows:
minimize Y
〈− log(Ẑ),Y 〉 (9)
subject to Y ᵀ1N = µ, Y 1K = 1N . (10) Extra constraint on Y are imposed simply cos the tuned estimation has to fulfil the basic probabilistic requirement that its sum is one. Comparing Eq. (9-10) with Eq. (4), we can see that if we substitute P with Y and substitute r and c with 1N and µ respectively, we find that the above optimization problem is actually a special case of OT.
In preliminaries, the entropy regularised OT is introduced (EOT), by adding entropy regularisation to OT, the given equation can be solved efficiently by Sinkhorn algorithm. Specifically, the optimisation problem is:
minimize Y
〈− log(Ẑ),Y 〉+ λY log(Y ) (11)
subject to Y ᵀ1N = µ, Y 1K = 1N . (12) The associated algorithmic flow is outlined in detail in Algorithm 1.
Algorithm 1: Solve OT-related algorithm efficiently in the post-hoc correction via Sinkhorn Algorithm. Input: Cost matrixM , trade-off parameter λ, max number of iterations NT , iteration number t,
error threshold , current error δ, two probability vectors r and c, |·| denotes the vector norm.
Result: Refined predictions Y 1 InitialiseK = e−M/λ,uold = 1N , v = 1K , t = 0; 2 while t ≤ NT and δ ≤ do 3 u = r./Kv; 4 v = c./Kᵀu; 5 σ = |uold − u|; 6 uold = u; 7 t = t+ 1; 8 end 9 Output Y = diag(u)Kdiag(v)
Assigning λ with 1, it is observed that we equate our objective function to the KL divergence, thus illustrating the extensive nature of our approach. DARP (Kim et al., 2020a) has previously applied it to long-tailed semi-supervised classification.
Remark We would like to illustrate the non-applicable scenarios of our method. Firstly, our method requires a large number of samples for evaluation. This is because if the batch size is small, we can not guarantee that the desired marginal distribution can be satisfied within the batch. In some online scenarios, the sample-wise correction method is more suitable. In addition, our method assumes that the marginal distribution is already known, and in the paper we assume in particular that it is consistent with a uniform distribution.

heading: 3.2 COST FUNCTION LEARNING VIA LINEAR MAPPING
text: Simple functions is likely to be sub-optimal for the real data sets; this suggests the design of a better cost function to better fit the long-tailed recognition problem. However, manually designed cost functions require expert knowledge in different domains. Thus, we propose to use a linear mapping to automatically learn the cost function, which relieves the need to configuration.
More specifically, for predictions Z̃ generated by Softmax operation via leveraging linear transformation matrixW ,W ∈ RK×K is learned so that the following objective function is minimised.
minimize Y ∈R
− 〈Y , log Z̃〉+ λ〈Y , logY 〉, (13)
Y 1K = 1N Y ᵀ1N = µ, Z̃nk = exp(W ᵀΦ (xn))k∑K k exp(W ᵀΦ (xn))k .
The resulting formula can be illustrated using a simple one-layer network of weight parameterW , together with an error function in Eq. (13). By normalising each row of W , each element can be understood as the probability value of confusing two classes, and somehow as a transition matrix between classes. We initialise W with an identity matrix and use small learning rate to learn W . One could also absorb the Logit Adjustment term − log p(ys) as the fixed bias parameter into the network. Motivated by this description, we can use a general gradient descent algorithm, such as SGD, to optimise the error function. Take into account the implementation in practice, the direct calculation of Eq. (13) can be done using the Sinkhorn iterations (Cuturi, 2013; Frogner et al., 2015; Peyré et al., 2019) in mini-batch training efficiently. Besides, We term the proposed method as OTLM (Optimal Transport via Linear Mapping), Figure 1 illustrates the overview of OTLM.
Compute the gradients w.r.t W With the optimisation in Algorithm 1, it was conducted on the overall data set with a comparatively large number of data. Quite a different scenario now, as minibatch training is more favoured when it comes to neural networks. For this reason, the optimisation workflow in Algorithm 1, as pointed by Peyré et al. (2019); Viehmann (2019), will have a problem of batch stablisation. To this end, we perform logarithmic transformation to step 3-4 in Algorithm 1.
logu = log r − log (Mv) = log r − logsumexp ( 1
λ M − log v
) (14)
log v = log c− log (Mᵀu) = log c− logsumexp ( 1
λ M − logu
) . (15)
with log-sum-exp operation logsumexp (x) = log( ∑ i expxi). Assuming convergence is achieved in the Sinkhorn’s loops, it is not necessary for us manually to compute the derivative of the loss function in Eq. (13) w.r.t W . As a matter of fact, as long as we make sure that the entire optimisation process is differentiable, modern deep learning libraries can then automatically implement end-toend derivations, as it allows us to differentiate between the results of Sinkhorn’s loops as a mere composition of elementary operations.
Performance degradation caused by imbalanced distributions can be addressed with meta-learning based methods (Ren et al., 2020; Wang et al., 2020a; Li et al., 2021). Recent works have illustrated neu-
ral networks can learn more meaningful representations from the validation data set {(xvn, yvn)} Nv n=1. Inspired by the methods based on meta learning, Here we also take a validation data set to optimise the parameterW , but with the still significant difference in that we require no labelling information, thus avoiding a large expanse of labeling samples.

heading: 4 EXPERIMENTS
text: In this section, we first conduct experiments comparing our approach versus extant post-hoc correction methods on three data sets, including CIFAR-100-LT (Cao et al., 2019), ImageNet-LT (Liu et al., 2019), and iNaturalist (Van Horn et al., 2018) with varying backbones. Finally, we empirically make a comparison of our algorithm with alternative cutting-edge long-tailed recognition methods. Observe that, for one thing, a plausible coupling of our algorithm to any training-aware long-tailed recognition method exists. As an illustration of the potency and strong generalisation of our approach, we conducted post-hoc correction of the pseudo-predictions of both methods, which are fairly typical for the training stage of all data sets: The first is based on the cross entropy (CE), and the other is the recently proposed RIDE (Wang et al., 2020b), which is based on multi experts.

heading: 4.1 DATA SETS AND BASELINES
text: The baseline methods and data set are briefly described here, with implementation details placed in the appendix. In our experiments, we assume that the marginal distribution of the test set is uniform due to the characteristics of the three data sets.
Baselines: We compare our methods with (i) post-hoc correction methods including Logit Adjustment (Menon et al., 2020), DARP (Kim et al., 2020a), (ii) state-of-the-art methods including Focal Loss (Lin et al., 2017), LDAM (Cao et al., 2019), BBN (Zhou et al., 2020), Balanced Softmax (Ren et al., 2020), Causal Norm (Tang et al., 2020), LADE (Hong et al., 2020), M2m (Kim et al., 2020b), Decouple (Kang et al., 2019), LFME (Xiang et al., 2020), RIDE (Wang et al., 2020b)
Long-tailed data set: We take experiments on three data sets including CIFAR-100-LT, ImageNetLT, and iNaturalist. We build the imbalanced version of CIFAR-100 by downsampling samples per class following the profile in Liu et al. (2019); Kang et al. (2019) with imbalanced ratio 10, 50, and 100. For all the benchmarks, we evaluate the performance on the test data set using a model trained on the training data set, and report the results using top-1 accuracy.

heading: 4.2 MAIN RESULTS
text: Table 1 illustrates the results on the CIFAR-100-LT data set. For CE-based training-aware methods, it first shows that Logit Adjustment is indeed a simple yet effective method for post-hoc correction. Remarkably, it outperforms the baseline methods by 2.6%, 4.1%, and 4.1% under the imbalance ratio of 10, 50, and 100 respectively. With a quick look at the results based on the OT approach, the superior results stress the advantages of our approach over the DARP and Logit adjustments. OT algorithm can further improve the performance by 0.62%, 0.43%, 0.70%, for OTLM, it outperforms
Logit Adjustment by 0.55%, 0.73%, 0.99% under the three imbalance ratios on CIFAR-100-LT data set. For RIDE-based training-aware methods, there is also a 0.6% improvement in the accuracy of our method.
Table 1 also provides the results on ImageNet-LT and iNaturalist data sets. The fact again attracts our attention that for CE-based training-aware methods, OT, which is based on convex optimisation, consistently outperform Logit Adjustment by a large margin (1.76%) on iNaturalist data set. The accuracy boosting indicates the great potential of methods based on optimisation for post-hoc correction. Besides, not surprisingly, OTLM can further enhance the prediction accuracy, it outperforms Logit Adjustment by 0.73% and 2.86% on ImageNet-LT and iNaturalist respectively. As for RIDE, the gain in accuracy is also significant, about 3% on iNaturalist.

heading: 4.3 COMPARISON WITH THE STATE-OF-THE-ART METHODS
text: Armed with a demonstration of the validity of OT and OTLM, we switched to a comparison of our performance with existing methods that achieved the most advanced results on three benchmarks. As shown in Table 2, in particular, on ImageNet-LT and iNaturalist data sets, the experimental results are remarkable and impressive, our method outperforms RIDE by 1.0% and 3.0%, respectively. Since iNaturalist is a very difficult and fine-grained data set consisting of 8,142 categories. These huge performance gains come at a fraction of the cost, as we show in the next subsection.

heading: 4.4 COMPUTATION COST OF OT AND OTLM
text: As we have highlighted, the additional computational cost of OT and OTLM is particularly small, compared to that of training-aware methods. In Table 3, exact time of the evaluations on the ImageNetLT and iNaturalist data sets is provided. Please note, the times here are measured from the start of each method to the final best performance. For the comparison between OT and OTLM, because
OTLM runs on the GPU, one optimization iteration of OTLM is in fact much faster than OT. Except for OTLM, which was run on an NVidia card (V100), the results come from a 28-core machine (2.20 Ghz Xeon). Firstly, we can observe that all post-hoc correction methods here are not quite time-consuming. For comparison, ImageNet-LT and iNaturalist can be trained on 4 NVidia cards (V100) for extremely long periods of time. Using ResNet-50-32x4d on the iNaturalist training data set, for example, with a batch size of 256, the amount of training time (in seconds) to perform 1 iteration is approximately 850. Also, if one differs the performance of each method, we gather that OT and OTLM provide the best balance of performance and running costs to match.

heading: 4.5 CONVERGENCE BEHAVIOR OF OTLM
text: To verify that our adopted OTLM algorithm is really efficient. With ResNet-50 as the backbone, we plotted curves depicting the decreasing absolute error σ with the number of iterations t on the ImageNet-LT validation data set. Image on the left in Figure 2 gives us a visualisation of the optimisation process. It is found that the absolute error σ converges at around 60 iterations. In fact, we discovered that 10 or even fewer iterations were enough to yield satisfactory performance.

heading: 4.6 DETAILED ANALYSIS ON DISTRIBUTION MATCHING
text: Upon distribution fitting, preferably the estimation bias is lessened, which implies that the KL divergence distance between the refined prediction and the target distribution is decreased. Table 4 displays the KL distances on the ImageNet-LT and iNaturalist data sets, and we can find that OT possesses a smaller KL distance than Logit Adjustment, which suggests that OT decreases the estimation bias more than Logit Adjustment by a much larger margin.

heading: 4.7 VISUALISATION ON WEIGHT PARAMETER IN OTLM
text: As a natural consequence, one wonders about the exact structure of this weight matrixW , and no better way to present its results than through visualisation. Towards this end, we applied OTLM
to refine the original predictions on CIFAR-100 data set with an imbalanced ratio R = 100 and ResNet-32 backbone, and trained a one-layer model until it converged to extract the value of W , where the shape of is 100 × 100. We visualised a sub-block of size 20 × 20, the illustration is presented on the right image in Figure 2. The very physical meaning of the elements inW can be interpreted as the probability of transition between classes. The visualised matrix is very dense, this non-sparsity suggests that the OTLM manages to correct the predictions of the model in the training stage, whilst minimising the distance between the desired and the actual distribution.

heading: 5 RELATED WORK
text: Long-tailed recognition: The great majority of current methods applied to long-tailed recognition can be divided into either training-aware and post-hoc correction methods. The training-aware methods aim to improve the generalisation ability of models trained on imbalanced training data set. Among them, re-sampling and re-weighting are two major methods. Re-sampling controls the class numbers by the means of over-sampling (Chawla et al., 2002) and under-sampling (Barandela et al., 2004; Drummond et al., 2003). The re-weighting approaches achieve balanced learning by giving smaller weights to majority classes and larger weights to minority classes (Tang et al., 2008; Zadrozny et al., 2003; Lin et al., 2017; Lee et al., 2017; Cui et al., 2019; Khan et al., 2019). Other approaches have also been proposed to resolve the imbalanced problem. For example, Cao et al. introduces a theoretically-principled loss function motivated by minimizing the generalisation error bound (Cao et al., 2019), Kang et al. proposes to decouple the learning phases into representation learning and classifier tuning (Kang et al., 2019). Apart from training-aware methods, post-hoc correction also offers us an avenue for long-tailed recognition problem, it can be applied to the normalisation of classifier weights (Kang et al., 2019; Zhang et al., 2019; Kim & Kim, 2020) or logits based on label frequency (Provost, 2000; Collell et al., 2016). Among them, Logit Adjustment (Menon et al., 2020) is a simple yet powerful methods to use. Compared with Logit Adjustment, our approach perform the post-hoc correction from the optimisation view.
Optimal transport: OT distances is a important family of distances for probability measures (Villani, 2008; Santambrogio, 2015). It has been extensively applied in different fields in machine learning. Courty et al. (Courty et al., 2016) proposes a regularised unsupervised optimal transportation model to perform the domain adaption. Other applications include 3D shape matching (Su et al., 2015), generative model (Arjovsky et al., 2017; Salimans et al., 2018; Bunne et al., 2019; Deshpande et al., 2019; Bhagoji et al., 2019), graph matching (Xu et al., 2019a;b), model designs Kandasamy et al. (2018); Chizat & Bach (2018). To the best of our knowledge, we are the first to perform the post-hoc correction from the OT perspective.

heading: 6 CONCLUSION
text: In this paper, we present two techniques for post-hoc corrections from an optimisation perspective. Our approach reviews the basic mathematical formulation of the distribution alignment problem and relates the resulting formulation to OT, which we find, moreover, offers greater possibilities and flexibility to solve the problem. In order to avoid manual configuration of the cost matrix, we propose to learn the cost matrix automatically by linear mapping, which leads to further performance improvements. The experimental results confirm the superiority of our proposed approach. I believe our work can open up more possibilities and imagination for post-hoc corrections.

heading: A OPTIMAL TRANSPORT ON LONG-TAILED SEMI-SUPERVISED CLASSIFICATION
text: Together, with the extension of our approach, we adapt it to semi-supervised long-tailed classification tasks. We bring our approach together with the advanced semi-supervised learning methods FixMatch Sohn et al. (2020). Adhering to the configuration in DARP Kim et al. (2020a), we also create synthetically long-tailed variants of CIFAR-100 with imbalanced ratio R = 20. The sample size of the unlabelled data is twice as large as the labelled data, with the same proportion of imbalance between them. To assess the performance, we report two popular metrics: balanced accuracy (bACC) Huang et al. (2016) and geometric mean scores (GM) Branco et al. (2016). Experimentally, with the exception of an OT to align the marginal distribution of the unlabelled data, all settings are consistent with DARP, and we set λ to 0.005. The results are reported in Table 5. It can be seen that our approach also achieves much better performance on semi-supervised learning tasks.
B PARAMETER STUDY OF λ
We conduct an additional experiment on iNaturalist data set to study the effect of λ on the final performance. The value of λ is selected from {0.01, 05, 0.1, 1.0.2.0, 5.0}, all other experimental settings are the same, including various hyperparameters, network structure, etc. As illustrated in Figure 3, we can find that OT achieves the best performance when λ = 1.
C IMPLEMENTATION DETAILS
The specific implementation details for each data set under the different methods are described below.
C.1 IMPLEMENTATION DETAILS OF CROSS ENTROPY
CIFAR-100-LT: During the training stage, following the recipe of (Tang et al., 2020; Hong et al., 2020), we apply SGD with batch size 256 and weight decay 0.0005 to train a ResNet-32 (He et al., 2016) model for 200 epochs, we employ the linear warm-up learning rate schedule for the first five epochs. We also set the base learning rate to 0.2 and reduce it at epoch 120 and 160 by a factor of
100. During the post-hoc stage, we use Adam with batch size 10k and weight decay 0.001 to train the single layered feed-forward network for 20 iterations. We set λ to 0.1, T to 200, ∆ to 0.001. The learning rate is constant and set to 0.001.
ImageNet-LT: During the training stage, following the implementation of (Hong et al., 2020), we train the ResNeXt-50-32x4d (Xie et al., 2017) for 90 epochs and perform a cosine learning rate scheme with an initial learning rate of 0.05. SGD is also employed to optimise the neural network with batch size 256, weight decay 0.0005, and momentum 0.9. During the post-hoc stage, we use SGD with batch size 10k, weight decay 0.001 to train the single layered feed-forward network for 100 iterations. The constant learning rate is 0.2. We set λ to 1.2, T to 7, ∆ to 0.001.
iNaturalist: During the training stage, ResNet-50 (He et al., 2016) is chosen as the backbone network. We use SGD with momentum 0.9, batch size 256 to train the network. We utilise the cosine learning rate schedule gradually decaying from 0.1 to 0. During the post-hoc stage, we use SGD with batch size 23,826, weight decay 0.001, λ to 1.0, ∆ to 0.001 and T to 10. The constant learning rate is set to 0.05. As iNaturalist has no test data set, we directly report the performance on validation data set.
C.2 IMPLEMENTATION DETAILS OF RIDE
The training details we use in RIDE share the same with the original paper, there are 6 experts in our RIDE on all three data sets.
CIFAR-100-LT We experimented with R = 100 and ResNet-32 as the base network. SGD is utilised as our solver with momentum 0.9, batch size 128, and epoch 200. The learning rate is initialised as 0.1 and decayed by 0.01 at epoch 120 and 160 respectively.
ImageNet-LT We take experiments with Re as backbone with 100 epochs. We set the initial learning rate to 0.1 and reduce it by 10 at epoch 60 and 80. SGD with batch size 256 and momentum 0.9 is also adopted as the optimisation solver here.
iNaturalist We conduct experiments with ResNet-50 as backbone. The training details are the same as ImageNet-LT except from batch size 512.

Review summaries:
This paper contributes an extension of post hoc correction of long-tailed recognition with Optimal Transport (OT). Unlike the previous work (e.g. logit adjustment) which focuses on sample-wise correction, this work, on the other hand, considers the marginal distribution of the overall data for correction. The method is further extended by learning a cost matrix. Their experiments show the effectiveness of optimal transport in the long-tail recognition problem via comprehensive comparison with previous works in terms of performance and efficiency.

This paper proposes a new method for post-hoc correction in long-tailed recognition. Specifically, they leverage the idea of optimal transport (OT) and propose a linear mapping to replace the original exact cost matrix in OT problem. From the experiments, the proposed method can be combined with existing methods and boost their performance further.