Title: Relative Entropic Optimal Transport: a (Prior-aware) Matching Perspective to (Unbalanced) Classification
Venue: NeurIPS 2023
Accepted: yes
Keywords: Optimal Transport; Unbalanced Classification
Abstract: Classification is a fundamental problem in machine learning, and considerable efforts have been recently devoted to the demanding long-tailed setting due to its prevalence in nature. Departure from the Bayesian framework, this paper rethinks classification from a matching perspective by studying the matching probability between samples and labels with optimal transport (OT) formulation. Specifically, we first propose a new variant of optimal transport, called Relative Entropic Optimal Transport (RE-OT), which guides the coupling solution to a known prior information matrix. We gives some theoretical results and their proof for RE-OT and surprisingly find RE-OT can help to deblur for barycenter images. Then we adopt inverse RE-OT for training long-tailed data and find that the loss derived from RE-OT has a similar form to Softmax-based cross-entropy loss, indicating a close connection between optimal transport and classification and the potential for transferring concepts between these two academic fields, such as barycentric projection in OT, which can map the labels back to the feature space. We further derive an epoch-varying RE-OT loss, and do the experiments on unbalanced image classification,  molecule classification, instance segmentation and representation learning. Experimental results show its effectiveness.

Relative Entropic Optimal Transport: a (Prior-aware) Matching Perspective to (Unbalanced) Classification

Abstract

Classification is a fundamental problem in machine learning, and considerable efforts have been recently devoted to the demanding long-tailed setting due to its prevalence in nature. Departure from the Bayesian framework, this paper rethinks classification from a matching perspective by studying the matching probability between samples and labels with optimal transport (OT) formulation. Specifically, we first propose a new variant of optimal transport, called Relative Entropic Optimal Transport (RE-OT), which guides the coupling solution to a known prior information matrix. We gives some theoretical results and their proof for RE-OT and surprisingly find RE-OT can help to deblur for barycenter images. Then we adopt inverse RE-OT for training long-tailed data and find that the loss derived from RE-OT has a similar form to Softmax-based cross-entropy loss, indicating a close connection between optimal transport and classification and the potential for transferring concepts between these two academic fields, such as barycentric projection in OT, which can map the labels back to the feature space. We further derive an epoch-varying RE-OT loss, and do the experiments on unbalanced image classification, molecule classification, instance segmentation and representation learning. Experimental results show its effectiveness.

1 Introduction

Long-tailed label distribution or unbalanced classification in training given more balanced testing set has been a challenge for existing neural models [42, 3, 62]. Various approaches have been developed, such as Class Re-balancing [3, 25, 35, 8, 45, 52], Information Augmentation [36, 7], and Module Improvement [65, 16, 62]. However, the problem is still far from being solved.

Most existing methods on (long-tailed) classification study the conditional probability function p ( y | x ) or their joint distribution given the label y and sample x , almost taking it for granted. For instance, [45] shows that in the long-tailed case, according to Bayes' theorem, the vanilla Softmax classification is affected by the label distribution shift, which makes the classifier more inclined to consider the samples as belonging to the majority class. To address this issue, the work explicitly considers the label distribution shift and re-derives the Softmax function. [22] argues that the conditional distribution p ( y | x ) is not the same between training and testing data, which can result in target-shift. To enhance class-unbalanced learning, the authors propose to relax the assumption that the source and target domains share the same conditional distribution p ( x | y ) . Despite the de-facto Bayesian perspective for understanding classification, in this paper, we take a matching view and demonstrate how new insights and approaches can be derived, especially for the challenging long-tail cases.

From the matching perspective, the problem becomes matching samples to labels. During training, the goal is to learn the distances between the features of samples and labels. During testing, matching results can be obtained by calculating the distances between the features of testing samples and their labels. We formulate this matching using the theory of Optimal Transport (OT), where training precisely involves learning the Inverse OT [12, 33, 51], and testing involves optimizing the transportation results. In the commonly studied long-tailed setting [35], the labels of training samples follow a long-tail distribution, while the labels are uniformly distributed for testing data. With our OT formulation, we demonstrate that these two different priors (i.e. long-tailed and uniform) can effectively reduce the difference between training and testing, thereby increasing the accuracy of the matching probability.

To incorporate the prior information e.g. the aforementioned typical long-tailed setting, we introduce the smoothing-guided prior matrix Q to vanilla entropic OT and call the resulting formulation relative entropic OT (RE-OT). To this new variant of OT, we provide convergence analysis, its corresponding static Schrödinger form, Sinkhorn algorithm for solving the problem as well as the dual formulation of RE-OT, which are mostly similar to those for the vanilla entropic OT. We find our RE-OT can help to deblur for barycenter images by setting specific smoothing prior Q .

Specifically, we develop the inverse RE-OT (as our relative entropy OT's counterpart to inverse OT) to learn the cost matrix or equivalently, the sample features for training data. And then we use the RE-OT to optimize the transportation to get the matching results (i.e. predictions for testing data). It is uncovered that many Softmax-based cross-entropy losses (e.g. Balanced Softmax, A-softmax and etc) are special cases of our derived new loss under half constraints of OT by specifying different forms of the cost matrix, supervision matrix in IOT and the smoothing-guided prior matrix Q . In other words, we can generalize the cross-entropy loss by specifying different cost matrix, smoothing-guided matrix, or other settings in the framework of inverse OT. We also find that when the regularization coefficient glyph[epsilon1] → 0 , the loss of inverse RE-OT equals triplet loss in metric learning fields. Additionally, our matching perspective reveals interesting connections between OT and classification tasks. For example, we discover the equivalence between the coefficient of entropic regularization in OT and the temperature in Softmax. We also find that the concept of barycentric projection in OT can be used in classification to calculate the class barycenters in hidden feature space. We believe that the connection between these two fields will inspire further algorithmic development for both sides. This paper contributes in the following ways:

1) We propose a matching perspective to revisit and provide new understanding to the classification problem with OT formulation. The matching perspective provide a clear interpretation to the challenge of long-tailed classification, which lies in the inconsistency of the utilized prior information between the matches learned from the training set and those from the testing set.

2) We propose a new variant of entropic optimal transport, called relative entropic OT (RE-OT), for learning matching with a specified prior. We provide theoretical results on various aspects of RE-OT, including its solution properties, Sinkhorn algorithm, barycenter calculation, and its dual form. As a side product of our study, we find that RE-OT can help to deblur barycenter images by setting a specific smoothing prior Q , which entropic OT alone cannot achieve.

3) We further develop Inverse RE-OT to link our results to classification, and show that Softmax and its variants are special cases of our derived loss based on Inverse RE-OT. This finding inspires a simple yet effective technique in practice for long-tailed classification: we let the smoothing-guided matrix Q to be epoch-varying, which gradually provides prior information and improves the performance.

4) Our study has revealed a significant link between Optimal Transport (OT) and existing classification tasks, such as the relationship between the regularization coefficient in OT and the temperature in Softmax. This suggests that OT concepts can be applied to classification tasks, and vice versa. For instance, the barycentric projection concept in OT can be utilized in classification tasks to calculate the class barycenters in the hidden feature space. We anticipate that this connection between the two fields will lead to further algorithmic development for both OT and classification.

5) Our method performs competitively and often outperforms baselines on various long-tailed benchmarks ranging from image classification, to molecule property prediction, and instance segmentation. Source code will be made publicly available.

2 Preliminaries and Related Work

Optimal Transport. Known as Wasserstein distance or Earth Mover's distance, Optimal transport (OT) [26], has gained significant attention in various fields including domain adaption [11], generative models [15, 34, 49], and image registration [14]. We begin by briefly introducing discrete Optimal Transport (OT), which provides a framework for viewing the (long-tailed) classification problem from a matching perspective. For a more detailed introduction to OT, readers are directed to [43].

Consider two histograms a ∈ Σ n and b ∈ Σ m , where the simplex Σ d = { x ∈ R d + | x glyph[latticetop] 1 d = 1 } . We can represent the transportation polytope U ( a , b ) , which is the polyhedral set of n × m matrices:

where 1 n and 1 m are n and m dimensional vectors of ones. U ( a , b ) contains all n × m nonnegative probabilistic matrices with row and column sums a and b . Then the Kantorovich's optimal transport problem can be defined as [26]:

where C ∈ R n × m + is the cost matrix for the distance between sample { x i } n i =1 and { y j } j m =1 . The above minimization is a linear program and the optimal solution P ∗ can be obtained with the network simplex [54] or other off-the-shelf techniques e.g. [40], which require significant time overhead. For its cost-effectiveness, the entropic regularization [57] is more popular and it gets an approximation:

where the regularizer is specified as H ( P ) = ∑ ij -P ij (log( P ij ) -1) . It can be proved that [43] when glyph[epsilon1] → 0 , the unique solution of Eq. 3 converges to the optimal P ∗ of original problem. And when glyph[epsilon1] → + ∞ , we can get

where a ⊗ b = a glyph[latticetop] b ∈ U ( a , b ) . So we can find the entropic regularization H ( P ) exactly do a smoothing to the solution towards a ⊗ b . Recently, Inverse Optimal Transport (IOT) start to be studied [12, 33, 51, 50] which assumes that the cost C is unknown for learning, given the established coupling. The inverse of entropic OT problem can be formulated as:

Specifically, [51] proposes a principled approach to infer the unknown costs and [6] develops the mathematical theory for IOT. [33] shows that by IOT, one can also modify and improve the raw matching prediction by the cost, which is learned by the raw matching as noise supervision.

In this paper, we develop an IOT-based formulation to learn the cost between samples and their labels, which further allows one to learn the feature extractor of the training samples. Then with the features exacted from testing data, we match them with target labels and achieve the goal of classification.

Long-tailed recognition. Long-tailed recognition has been a standing problem with wide attention in vision and learning [64, 19, 35]. One common approach [45, 41] is to re-balance the class distribution in the training set. For example, some works employ re-sampling [25] or re-weighting [8] techniques to let the model pay more attention to the minority classes in the training set. Data augmentation is also adopted to synthesize more samples for the underrepresented classes [28]. While in [9], transfer learning is used, which leverages knowledge learned from a source domain with sufficient data to improve the performance on a target domain with imbalanced data. This line of research includes techniques such as fine-tuning [63], domain adaptation [53], and meta-learning [56]. Recently, some works have proposed to explicitly model the long-tailed distribution, such as by adjusting the loss or designing specialized network architectures. Examples include focal loss [35], class-balanced loss [8], balanced Softmax loss [45]. In this paper, we view classification from a matching perspective. Our method does not predict the label or class directly but learns the features in training and calculates the matching probability instead. With the aid of OT, we formulate it with relative entropic as proposed in the next section, which adopts different prior for training and testing.

0.10

0.08

90'0-

- 0.04

0.02

0.00

{Π ∥⊕

〈Ψ

)GLYPH<228> 〈Ψ〈Ω

{Π ∥⊕

{Π ∥⊕

〈Ω

{Π ∥⊕

〈Ψ

)GLYPH<228> 〈Ω

∥Ø

{︸

3 Relative Entropy Regularization for Optimal Transport

Motivation of introducing prior into OT. As introduced in Eq. 3, entropic regularization (ER) is an effective method [10] for approximating the Kantorovich OT solution. It pushes the original linear programming solution away from the hard boundary and obtains a smoother solution by minimizing a glyph[epsilon1] -strongly convex function. With the increase of the penalty coefficient, the solution progressively approaches a more 'smooth' solution, ultimately converging to a glyph[latticetop] b as illustrated by Eq. 4. However, a question is raised: is such a smoothing direction a glyph[latticetop] b appropriate for all practical situations, e.g. the long-tailed problems? Can we freely choose a more suitable (smoothing) direction to achieve more effective results tailored to the problem or dataset at hand? In this section, we will develop relative entropic OT (RE-OT), a more general formulation than traditional entropic OT [57].

Formulation. Given a cost matrix C ∈ R n × m + and two margins a ∈ Σ n and b ∈ Σ m , where Σ n and Σ m represent the probability simplex with n and m bins, respectively, we introduce the positive smoothing-guidance matrix Q as a manually-specified constant prior to guide the smoothing. We use the relative entropy regularizer H Q ( P ) in place of the traditional entropy in Eq. 3 to achieve this

The relative entropy H Q ( P ) can increase the probability of transportation for the element P ij if Q ij is larger. Consequently, if the prior in the form of Q is available, the use of relative entropy regularization can improve the efficiency of learning the transportation solution. Therefore, for long-tailed tasks, we can assign a large Q element to the majority classes and a low element of Q to the tailed classes to enhance training. We will discuss this in detail in the next section.

For Eq. 6, we further study its properties. Firstly, we show the relation between RE-OT and the entropic OT by Eq. 3 as follows. The proof is given in Table A.

Proposition 1 ( Solution Propriety. ) . Assume P glyph[epsilon1] Q , P glyph[epsilon1] and are the optimal solution of RE-OT and entropic regularized OT, respectively. Then we can get:

(1) When glyph[epsilon1] → + ∞ , the optimal RE-OT's solution P glyph[epsilon1] Q will converge to ˜ Q where ˜ Q takes the form ˜ Q = diag ( u ) Q diag ( v ) with two uniquely defined non-negative vectors u and v .

(2) With the prior Q and its corresponding ˜ Q as defined in (1), we have P glyph[epsilon1] Q = P glyph[epsilon1] ˜ Q . And when ˜ Q = a ⊗ b , we have the equality P glyph[epsilon1] = P glyph[epsilon1] .

Proposition 1 shows the relation between RE-OT and entropic OT, and we can find RE-OT is a more general formulation. Note ˜ Q is defined by Q and can be obtained by iteratively normalizing row/column sums of Q . Fig. 1 shows the results of P glyph[epsilon1] Q by varying glyph[epsilon1] . As glyph[epsilon1] increases, the optimal coupling becomes more dense, but P glyph[epsilon1] becomes more uniform while P glyph[epsilon1] Q converges to ˜ Q .

Besides, similar to the standard entropic regularized OT, our RE-OT in Eq. 6 can also be reformulated to the 'static Schrödinger problem' form [31], which exactly leans a projection under KL divergence.

Proposition 2 ( static Schrödinger form ) . Redefine a general KL divergence as

The optimization in Eq. 6 is equivalent to the following minimization, where K ij = Q ij e -C ij /glyph[epsilon1] :

The proof is in Appendix B. Prop. 2 shows the optimal solution P glyph[epsilon1] Q is exactly the KL projection of K onto U ( a , b ) . In fact, K can be set as K ij = ˜ Q ij e -C ij /glyph[epsilon1] due to the equivalence between P glyph[epsilon1] Q and P glyph[epsilon1] ˜ Q . Expect for the different form of K , RE-OT and entropic OT share the same static Schrödinger formulation, which means that many methods in entropic OT can be applied to RE-OT, such as iterative Bregman projections [2].

The Sinkhorn algorithm and barycenters for RE-OT. The optimal solution for RE-OT can be estimated using the Sinkhorn algorithm, as its counterpart has already been well-developed for solving entropic OT [10]. The algorithm for RE-OT shares the same form as that for entropic OT, given K ij = ˜ Q ij e -C ij /glyph[epsilon1] . Specifically, with two non-negative vectors u ′ and v ′ uniquely defined up to a multiplicative factor, the optimal solution has the form P glyph[epsilon1] Q = diag( u ′ ) K diag( v ′ ) , which can be efficiently computed by iterating u ′ , v ′ ← a ./ Kv ′ , b ./ K glyph[latticetop] u ′ . The proof and algorithm are presented in Appendix C. In addition, the barycenter between distributions is a natural extension of OT [1, 2], and with the matrix K , [2] defined it with a weighted KL projection problem:

where { λ s } S s =1 are the weights, b s are known histograms representing images, and a is the barycenter histogram to be calculated. Though easier to calculate compared with the regularized formulation in [1], a drawback of this entropic-based method is that it can lead to blurred barycenters, and methods have been proposed to address this issue [23]. By calculating barycenters between noise and an image (i.e. S = 2 here), we show that the blurred problem can be simply solved by setting λ where λ 1 = λ, λ 2 = 1 -λ and the prior matrix Q :

where P glyph[epsilon1] is the optimal solution of entropic OT from image b 1 to b 2 . Fig. 2 illustrates the effect of different choices of Q when computing barycenters between noise and an image. Without using Q , the resulting barycenters are very blurry. When we set Q to be P glyph[epsilon1] , the barycenters display the shape of the leopard clearly, but fail to transition smoothly from noise to leopard image as λ changes. To address this issue, we set Q as shown in Eq. 10, which can blur the image while transitioning smoothly with the gradient.

Proposition 3 ( Dual formulation ) . From the optimization in Eq. 6, we can get its dual formulation:

where f ∈ R n and g ∈ R m are the corresponding dual variables.

The proof is given in Appendix D. Exactly f , g are linked to u , v appearing in Sinkhorn algorithm by u = exp( f /glyph[epsilon1] ) and v = exp( g /glyph[epsilon1] ) , and thus Sinkhorn algorithm can be done in log-domain [47].

Setting Q with the Optimal Solution Iteratively. The intuition for the prior is to iteratively update the solution P glyph[epsilon1] Q as a new Q , i.e., Q ( t ) = P glyph[epsilon1] Q ( t -1) , where Q ( t ) represents the solution obtained after the n -th iteration. The question then arises as to how Q ( t ) changes over time. We find that this problem is equivalent to a proximal point algorithm:

Here F ( Q ) = < C, Q > + l U ( a , b ) ( Q ) , where l U ( a , b ) ( Q ) is defined such that l U ( a , b ) ( Q ) = 0 if Q ∈ U ( a , b ) , and l U ( a , b ) ( Q ) = + ∞ otherwise. We find this problem is discussed by [29, 43],

0

Q

g

a

=0

=1/5

=2/5

=3/5

=4/5

=1

which aims to get the optimal solution of the non-regularized OT. As discussed in Sinkhorn of RE-OT, we have the optimal solution form:

When we set Q (0) = 1 n × m , the optimization is equivalent to applying Sinkhorn's algorithm iteratively with a kernel e -( t +1) C glyph[epsilon1] , i.e., with a decaying regularization coefficient glyph[epsilon1] t +1 . This observation further emphasizes the importance of selecting an appropriate Q for RE-OT.

4 RE-OT for Long-tailed Recognition

We now discuss the application of inverse REOT to address the long-tailed classification task (including the contrastive learning setting). Unlike previous works that study this problem using conditional probability, our approach aims to learn features by matching samples with their labels using Inverse OT. Specifically, given the mini-batch pair set { ( x i , y i ) } n i =1 where x i is training sample and y i is the m -dimensional one-hot label of x i . We can set the the cost C θ ij between sample x i and another one-hot label given the neural network f θ . Without loss of generality, we can set C θ ij = c -l ij where l ij = ( f θ ( x i )) j is the j -th component of logits

TrainingData

Cost

Coupling

Supervision

Matrix

√个

P

Learning the Inverse RE-OT

labelfeatures

Prior Q

(e.g.one-hot)

featureextractor

OptimizingtheRE-OT

Cost

Predic

→Learning

Matrix

tion

TestingData

Inference

f θ ( x i ) for sample x i . Then we can learn the matching between features and labels with inverse OT via a bi-level optimization:

where ˜ P is the supervision given the label { y i } . For example, we can set ˜ P ij = y ij when y ij is the j -th component of y i . Here U refer to the constraints for the coupling P . We can set U = U ( a , b ) with the full matching constraints or its relaxation e.g. U = U ( a ) = { P | P1 m = a } .

For our designed smoothing guidance Q , a simple idea is to set it with the long-tailed ratio i.e. the training sample ratio of class labels which are readily available from the training set, which equals to

the Balanced Softmax method proposed in [45]. In this paper, motivated by Eq. 10 having a good efficiency in OT and other works [60] which adopt two stages for learning, we let it vary over epochs:

where t is the training epoch and r is the prior probability matrix e.g. the balanced ratio in [45], teacher-based prior predictions or the sample-class-wise setting discussed in Appendix E. λ ( t ) is an epoch varying weight. When t is small, λ ( t ) is close to 0, and Q is more likely to be a uniform distribution, which gives the model less prior information. As t approaches the final training epoch number T , λ ( t ) → 1 and Q → r , which means the model is given the full prior information. This gradual process of introducing prior information can be helpful for training the inverse OT. We provide a specific setting for λ ( t ) and r in Appendix E. During the inference process, the problem reduces to the inner optimization in Eq. 6, which takes the form of traditional unsupervised OT. We use this optimization to obtain the matching between the sample features and labels, which corresponds to classification for the testing data. We do not adopt the long-tailed setting of Q during inference because the testing data is assumed to be uniform.

4.1 Generalization to a Family of Losses in Form of Softmax-based Cross-Entropy

We find our new formulation with inverse OT is a more general form if we set U = U ( a ) where a i = 1 /n and many Softmax-based cross-entropy losses can be our special case by varying the form of ˜ P , C θ and Q . Specifically, with U = U ( a ) , Eq.14 equals to

The proof is given in Appendix F. We can find the form in Eq. 16 is similar with the Softmax cross-entropy loss. In particular, if we set ˜ P , C θ , Q and glyph[epsilon1] in different forms as shown in Tab. 1, many classification loss can be the special cases of our formulation, as discussed in detail as follows.

Special cases by varying C θ . For a matching problem, it is quite important to define a distance among samples. We first set C θ ij = c -l ij where c is a large enough value. In this case, Eq. 16 reduces to a normal softmax-based cross-entropy loss when Q is in the uniformly distributed. Alternatively, we can set C θ differently to suit specific tasks. For example, in learning in the hypersphere space, the Vanilla Softmax will be converted to A-softmax [37] to enforce angular margin constraints for improved face recognition performance.

Deriving more losses by varying Q . By varying the setting of Q , we can derive many Softmaxbased losses as special cases of the RE-OT-based loss. For example, the Balanced Softmax can be obtained by setting Q ij = n j , where n j is the frequency of label j in the training set. This choice of Q encourages the model to assign equal weights to each class during training, which helps to mitigate the impact of class imbalance.

Discussion on ˜ P . Re-weighting the example is a common way to improve the Long-tailed learning, including the empirical Re-weighting methods (e.g. focal loss [35]) and automatic Re-weighting ways (e.g. L2RW [46]). In the view of IOT, the supervision of the coupling ˜ P is self adapting and may depend on P θ , which may be helpful in IOT application fields such as deep graph matching.

√⌈∐{˜({[˜∐̂}

̂∐√({[˜∐̂}

̂]√̂({[˜∐̂}

[}√√˜({√∐]⌈}

√[]√({√∐]⌈}

√√√̂⌋({√∐]⌈}

√√∐]{

⋃}˜√⌉∐glyph[arrowtp]

√˜√√

⊔}̂∐⌈

⊗√√√

̂∐√glyph[arrowbt]̂˜{√˜√

Triplet loss for glyph[epsilon1] → 0 + . Though Eq. 16 has a very different form compared to triplet loss, the equivalence can also be established when the temperature glyph[epsilon1] → 0 + . Note that the idea of the relation between triplet loss and our loss is motivated by the work [55] which gives the proof of the equivalence between triplet loss and InfoNCE. We give the proof in Appendix G.

Representation Learning with our Formulation Our matching perspective can be applied not only to unbalanced classification but also to unbalanced representation learning, in which we only need to modify the original one-hot label to the augmentation's features. In Sec. 5, we conducted experiments on long-tailed contrastive learning and verified that by selecting an appropriate Q , we can improve the effectiveness of representation learning on long-tailed datasets.

4.2 Bridging Regularized OT to Classification

We have shown the classification can be viewed as a matching problem and if we formulate the matching with optimal transport, we can get the equivalence between Softmax (as well as its variants) and RE-OT under the setting of U ( a ) . We can observe that there are some interconnected concepts between OT and classification, which can shed lights on a deeper understanding of OT and in turn the classification problem with different forms. One notable example is the entropy regularization coefficient glyph[epsilon1] in OT and the temperature coefficient τ in Softmax. There have been numerous works [24, 61] exploring the role of the temperature hyperparameter in Softmax. We can understand it easier from the view of OT, where glyph[epsilon1] serves as a relaxation hyperparameter for couplings yielding smoother predictions. In fact, there are other concepts, such as barycentric projection in OT [13], which may help us understand more about classification. We give its definition here:

Definition 1 ( Barycentric Projection ) . Consider the setting of Eq. 6 in which we use entropic regularization to approximate OT between discrete measures. One can define the so-called barycentric projection map with the coupling P

where { x i } ⊂ X is the set of locations corresponding to simplex a and { y j } is the set of locations in the Y space croponding to simplex b .

From the classification view, exactly T ( x i ) is probability confidences of samples x i which can be understand as the feature in the one-hot label space. Besides, motivated by this barycentric projection in OT, we can exactly define the mapping for the feature of one-hot label in hidden feature space:

where f i is the feature of x i extracting from the neural networks and F is the feature space. Eq. 18 maps the (one-hot) label to the feature space, which means no need to learn the centroid features

as done in oneline clustering methods [4, 32] but calculate it statically with Eq. 18 if the coupling is known. As shown in Fig. 4, we first train the models using vanilla softmax, focal loss, and our loss on the Cifar10-LT dataset. We consider the logits as the sample features and save the logits from the three head and three tailed classes, along with the corresponding predicted couplings, as matrices. Then, we calculate the barycenters of the labels using Eq. 18, which effectively computes the weighted average of the features. We concat the logits and barycenters as a new matrix and use it to calculate t-SNE results for training data, as shown in the first row of Fig. 4. For testing data, we concat the testing logits with the barycenters calculated from the training data and use t-SNE dimensional reduction, as shown in the second row in Fig. 4. By comparing the first row and the second row of Fig. 4, we can observe the differences in the positions of the barycenters, which are actually caused by shift in the feature distributions.

Further Discussions between OT and classification. Note that many theoretical aspects and variations of OT can be incorporated into the field of classification. Here are a few advantages unique to the OT perspective that can be achieved but are not possible in the traditional Bayesian view: 1) Variable constraints instead of { P : P1 = a } , such as (modified) Optimal Partial Transport (OPT) with { P : P1 ≤ a , 1 glyph[latticetop] P1 = s } where s is the number requiring to predict. This allows for distributions that are not just one-vs-all and enables rejection of classification if the model determines that a sample cannot be classified into any of the candidate labels; 2) Generalization of softmax. The current softmax formulation is essentially based on Entropic Regularization of OT. However, OT regularization goes beyond just the entropic regularization and includes other regularizations such as L2, Tsallis entropies, or divergence-based ones. This opens up possibilities for generalizing softmax; 3) Classification cross-representation models. From the OT perspective, classification between the features of samples and labels, based on Gromov-Wasserstein Distance, can be performed in different feature spaces. For example, the sample features may be in 100 dimensions while the label features are in 20 dimensions. This can be helpful for preserving privacy as Gromov-Wasserstein Distance [44] only requires similarity between samples and labels. Due to the various variants and theories within OT, considering classification as OT naturally allows us to incorporate existing OT knowledge into the field of classification, which is the ongoing work we are conducting.

5 Experiments

We evaluate our proposed approach for both image and molecule datasets. As the primary focus of this paper is the impact of loss functions on model learning, we do not adopt additional specialized techniques to improve performance, such as transfer learning or resampling. This is to control variables and thus all the baselines used in this study represent only the loss functions proposed by them. Additional experimental settings and details can be found in Appendix H.

Experiments on long-tailed image classification. The long-tailed image classification datasets analyzed in this study include CIFAR10-LT, CIFAR100-LT [30], and Imagenet-LT [38]. To evaluate the performance of the models, we use the corresponding balanced test dataset and report top-1 accuracy. Specifically, for CIFAR-10, we report accuracy on two sets of classes: Many-shot (more than 100 images) and Few-shot (less than 100 images). For CIFAR-100 and Imagenet, we report accuracy on three sets: Many-shot (more than 100 images), Medium-shot (20 ∼ 100 images), and Few-shot (less than 20 images). The experiments for unbalanced image classification are all conducted with an imbalanced factor of less than 200, which is defined as the ratio of the number of training instances in the largest class to the smallest [45]. The results for long-tailed classification are presented in Table 3. Note that in our approach, the teacher-based method involves setting the prior matrix r in Eq. 15 using a vanilla softmax trained teacher model. On the other hand, the ratio-based method refers to setting r as the long-tailed ratio. From a comprehensive perspective, at least one of our approaches achieves the best average accuracy across the entire dataset.

Experiments on unbalanced molecule classification. We further examine the application of our proposed method in the context of molecular representation learning task, which holds significant implications for healthcare [59] and drug discovery [17, 58]. Specifically, we perform the experiments on two unbalanced molecule classification datasets, i.e. OGBG-MOLBBBP and OGBG-MOLBACE, from the Open Graph Benchmark (OGB) [21]. We use the default train/val/test split with ratio 8:1:1. Given the presence of class imbalance within the two datasets, we have chosen to adhere to the precedent set by prior studies, also employing the ROC-AUC as the metric for performance evaluation. As the experimental results are easily influenced by initialization, we conducted five repetitions of

the experiment and reported the mean accuracy and its standard deviation in the Tab. 2, which show the effectiveness of our method.

Experiments on LongTailed Instance Segmentation. We do experiments on long-tailed instance segmentation with LVIS v1.0 datasets [18], as one of the most challenging datasets in vision with a much higher imbalance factor compared to the rest. We use the official splits and evaluation is conducted on validation

set. The setting mainly follows the experiments in [45], whose details are given in Appendix H. AP m , AP r , AP c , AP f are reported in Tab. 2. AP m denotes Average Precision of masks, AP r , AP c , AP f denote Average Precision of masks on rare classes, common classes and frequent classes. As shown in Tab. 2, we can find that our method achieves the best AP m on the entire set and perform competitively for other evaluations.

More Experiments. Wealso perform experiments on Long-Tailed contrastive representation learning on CIFAR10-LT, CIFAR100-LT with 100 imbalanced factor. We compare our methods with the methods in [5, 27]. The settings and results are given in Appendix H.5. Besides, to further demonstrate the usefulness of viewing classification as OT, a new inference method is proposed for testing. we replaced the softmax (i.e. constraints within U ( a ) ) with Sinkhorn algorithm (i.e. constraints within U ( a , b ) ), where b represents the assumed ratio in the testing data (e.g., long-tailed, uniform, or reverse long-tailed distribution). Details are given in Appendix H.6.

6 Conclusion

We have provided a matching perspective to provide new understanding to classification. Under our matching-based framework, we show that the inherent challenge of long-tailed classification lies in the inconsistency of the prior information used for training and testing under the matching formulation. We then develop inverse relative entropic OT approach to revisit the classification problem, and especially for the long-tailed case, we develop a simple yet effective technique based on our theoretical insights. Experimental results verify the effectiveness of our methods.

Broader impacts. Our work establishes a theoretical framework for understanding and solving (self-)supervised problems using OT theory. This not only broadens the range of applications for OT in generative models, such as WGAN and its variants, but also provides a new perspective for redefining, understanding, and solving (self-)supervised problems. Furthermore, the theory and concepts of OT can facilitate the discovery of interesting relationships and insights for supervised problems, such as the connection between open set recognition and unbalanced OT. Limitations. It assumes known prior label distribution testing data, which is often unknown in real-world scenarios.

References

M. Agueh and G. Carlier. Barycenters in the wasserstein space. SIAM Journal on Mathematical Analysis , 43(2):904-924, 2011.

J.-D. Benamou, G. Carlier, M. Cuturi, L. Nenna, and G. Peyré. Iterative bregman projections for regularized transportation problems. SIAM Journal on Scientific Computing , 37(2):A1111A1138, 2015.

K. Cao, C. Wei, A. Gaidon, N. Arechiga, and T. Ma. Learning imbalanced datasets with label-distribution-aware margin loss. Advances in neural information processing systems , 32, 2019.

M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems , 33:9912-9924, 2020.

T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

W.-T. Chiu, P. Wang, and P. Shafto. Discrete probabilistic inverse optimal transport. In International Conference on Machine Learning , pages 3925-3946. PMLR, 2022.

J. Cui, Z. Zhong, S. Liu, B. Yu, and J. Jia. Parametric contrastive learning. In Proceedings of the IEEE/CVF international conference on computer vision , pages 715-724, 2021.

Y. Cui, M. Jia, T.-Y. Lin, Y. Song, and S. Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9268-9277, 2019.

Y. Cui, Y. Song, C. Sun, A. Howard, and S. Belongie. Large scale fine-grained categorization and domain-specific transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4109-4118, 2018.

M. Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems , 26, 2013.

B. B. Damodaran, B. Kellenberger, R. Flamary, D. Tuia, and N. Courty. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In Proceedings of the European conference on computer vision (ECCV) , pages 447-463, 2018.

A. Dupuy, A. Galichon, and Y. Sun. Estimating matching affinity matrix under low-rank constraints. arXiv preprint arXiv:1612.09585 , 2016.

S. Ferradans, N. Papadakis, G. Peyré, and J.-F. Aujol. Regularized discrete optimal transport. SIAM Journal on Imaging Sciences , 7(3):1853-1882, 2014.

J. Feydy, B. Charlier, F.-X. Vialard, and G. Peyré. Optimal transport for diffeomorphic registration. In Medical Image Computing and Computer Assisted Intervention- MICCAI 2017: 20th International Conference, Quebec City, QC, Canada, September 11-13, 2017, Proceedings, Part I 20 , pages 291-299. Springer, 2017.

I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville. Improved training of wasserstein gans. In NIPS , 2017.

H. Guo and S. Wang. Long-tailed multi-label visual recognition by collaborative training on uniform and re-balanced samplings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15089-15098, 2021.

Z. Guo, W. Yu, C. Zhang, M. Jiang, and N. V. Chawla. Graseq: graph and sequence fusion learning for molecular property prediction. In Proceedings of the 29th ACM international conference on information & knowledge management , pages 435-443, 2020.

A. Gupta, P. Dollar, and R. Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5356-5364, 2019.

H. He and E. A. Garcia. Learning from imbalanced data. IEEE Transactions on knowledge and data engineering , 21(9):1263-1284, 2009.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770778, 2016.

W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems , 33:22118-22133, 2020.

M. A. Jamal, M. Brown, M.-H. Yang, L. Wang, and B. Gong. Rethinking class-balanced methods for long-tailed visual recognition from a domain adaptation perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7610-7619, 2020.

H. Janati, M. Cuturi, and A. Gramfort. Debiased sinkhorn barycenters. In International Conference on Machine Learning , pages 4692-4701. PMLR, 2020.

E. Jang, S. Gu, and B. Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144 , 2016.

B. Kang, S. Xie, M. Rohrbach, Z. Yan, A. Gordo, J. Feng, and Y. Kalantidis. Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv:1910.09217 , 2019.

L. Kantorovich. On the transfer of masses (in russian). In Doklady Akademii Nauk , volume 37, pages 227-229, 1942.

P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan. Supervised contrastive learning. Advances in neural information processing systems , 33:18661-18673, 2020.

J. Kim, J. Jeong, and J. Shin. M2m: Imbalanced classification via major-to-minor translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 13896-13905, 2020.

J. J. Kosowsky and A. L. Yuille. The invisible hand algorithm: Solving the assignment problem with statistical physics. Neural networks , 7(3):477-490, 1994.

A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.

C. Léonard. From the schrödinger problem to the monge-kantorovich problem. Journal of Functional Analysis , 262(4):1879-1920, 2012.

J. Li, P. Zhou, C. Xiong, and S. C. Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint arXiv:2005.04966 , 2020.

R. Li, X. Ye, H. Zhou, and H. Zha. Learning to match via inverse optimal transport. Journal of machine learning research , 20, 2019.

Y. Li, Y. Mo, L. Shi, and J. Yan. Improving generative adversarial networks via adversarial learning in latent space. Advances in Neural Information Processing Systems , 35:8868-8881, 2022.

T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision , pages 2980-2988, 2017.

J. Liu, Y . Sun, C. Han, Z. Dou, and W. Li. Deep representation learning on long-tailed data: A learnable embedding augmentation perspective. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2970-2979, 2020.

W. Liu, Y. Wen, Z. Yu, M. Li, B. Raj, and L. Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 212-220, 2017.

Z. Liu, Z. Miao, X. Zhan, J. Wang, B. Gong, and S. X. Yu. Large-scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2537-2546, 2019.

A. K. Menon, S. Jayasumana, A. S. Rawat, H. Jain, A. Veit, and S. Kumar. Long-tail learning via logit adjustment. arXiv preprint arXiv:2007.07314 , 2020.

J. Orlin. A faster strongly polynomial minimum cost flow algorithm. In Proceedings of the Twentieth annual ACM symposium on Theory of Computing , pages 377-387, 1988.

S. Park, J. Lim, Y. Jeon, and J. Y. Choi. Influence-balanced loss for imbalanced visual classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 735-744, 2021.

H. Peng, M. Sun, and P. Li. Optimal transport for long-tailed recognition with learnable cost matrix. In International Conference on Learning Representations , 2022.

G. Peyre and M. Cuturi. Computational optimal transport. Foundations and Trends in Machine Learning , 11(5-6):355-607, 2019.

G. Peyré, M. Cuturi, and J. Solomon. Gromov-wasserstein averaging of kernel and distance matrices. In International conference on machine learning , pages 2664-2672. PMLR, 2016.

J. Ren, C. Yu, X. Ma, H. Zhao, S. Yi, et al. Balanced meta-softmax for long-tailed visual recognition. Advances in neural information processing systems , 33:4175-4186, 2020.

M. Ren, W. Zeng, B. Yang, and R. Urtasun. Learning to reweight examples for robust deep learning. In International conference on machine learning , pages 4334-4343. PMLR, 2018.

B. Schmitzer. Stabilized sparse scaling algorithms for entropy regularized transport problems. SIAM Journal on Scientific Computing , 41(3):A1443-A1481, 2019.

F. Schroff, D. Kalenichenko, and J. Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 815-823, 2015.

L. Shi, Y. Li, and J. Yan. Iid-gan: an iid sampling perspective for regularizing mode collapse. arXiv preprint arXiv:2106.00563 , 2021.

L. Shi, G. Zhang, H. Zhen, J. Fan, and J. Yan. Understanding and generalizing contrastive learning from the inverse optimal transport perspective. International conference on machine learning , 2023.

A. M. Stuart and M.-T. Wolfram. Inverse optimal transport. SIAM Journal on Applied Mathematics , 80(1):599-619, 2020.

J. Tan, C. Wang, B. Li, Q. Li, W. Ouyang, C. Yin, and J. Yan. Equalization loss for long-tailed object recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11662-11671, 2020.

E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7167-7176, 2017.

G. R. Waissi. Network flows: Theory, algorithms, and applications, 1994.

F. Wang and H. Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2495-2504, 2021.

Y.-X. Wang, D. Ramanan, and M. Hebert. Learning to model the tail. Advances in neural information processing systems , 30, 2017.

A. G. Wilson. The use of entropy maximising models, in the theory of trip distribution, mode split and route split. Journal of transport economics and policy , pages 108-126, 1969.

N. Yang, K. Zeng, Q. Wu, X. Jia, and J. Yan. Learning substructure invariance for out-ofdistribution molecular representations. In Advances in Neural Information Processing Systems , 2022.

N. Yang, K. Zeng, Q. Wu, and J. Yan. Molerec: Combinatorial drug recommendation with substructure-aware molecular representation learning. In Proceedings of the ACM Web Conference 2023 , pages 4075-4085, 2023.

S. Zhang, Z. Li, S. Yan, X. He, and J. Sun. Distribution alignment: A unified framework for long-tail visual recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2361-2370, 2021.

X. Zhang, F. X. Yu, S. Karaman, W. Zhang, and S.-F. Chang. Heated-up softmax embedding. arXiv preprint arXiv:1809.04157 , 2018.

Y. Zhang, B. Hooi, L. Hong, and J. Feng. Self-supervised aggregation of diverse experts for test-agnostic long-tailed recognition. arXiv preprint arXiv:2107.09249 , 2021.

Y. Zhang, B. Hooi, D. Hu, J. Liang, and J. Feng. Unleashing the power of contrastive selfsupervised visual models via contrast-regularized fine-tuning. Advances in Neural Information Processing Systems , 34:29848-29860, 2021.

Y. Zhang, B. Kang, B. Hooi, S. Yan, and J. Feng. Deep long-tailed learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

B. Zhou, Q. Cui, X.-S. Wei, and Z.-M. Chen. Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9719-9728, 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an inverse Relative Entropic Optimal Transport (RE-OT) point of view for classification problems. The paper then proposes to use inverse RE-OT with a time-varying prior for solving long-tailed classification problems. Evaluations show improvements compared to existing baselines such as vanilla softmax and balanced softmax on long-tailed tasks.\n\nSTRENGTHS:\nThe reviewer believes the main strengths of the paper are solid theory and novelty of the framework. Specifically, \n\n- The OT and novel RE-OT theory is introduced clearly, with provided visualizations and explanations to aid understanding. \n- The novel RE-OT theory makes a nice connection and a unified view for many existing classification methods, as summarized in Table 1.\n- Experiment results show some advantage compared to existing methods.\n\nWEAKNESSES:\nThe reviewer mainly has concerns with the impact of the work. In particular, \n\n- Some additional intuitions would be helpful for the reviewer on why RE-OT gives a good or useful way for tackling classification problems. Currently, the RE-OT framework is laid out clearly, and the authors proposed to apply RE-OT to solve long-tailed classification, but the paper does not make clear what methodological or practical benefit the proposed framework has. \n- Comparing different methods in Table 1, the newly proposed method only differs from Balanced-Softmax in the choice of Q slightly, and the proposed choice of Q is not properly motivated in my opinion. The work also does not compare with using epoch-varying losses or two-stage approaches for other methods. \n- The evaluations show relatively minor improvements compared to existing works, and most experiments do not report error bars so it is unclear if most improvements are significant. \n\nThe work also has some minor issues with clarity:\n- The work uses $n$ for both number of iterations and dimension of polytope $\\Sigma_n$ which can be confusing. \n- Proposition 3 appears to be misplaced. \n- The notations in Figure 2 and equation (10) are quite hard to understand.\n\nQUESTIONS:\n- How does (15) relate to (10)? \n- Are (17) and (18) unique to the OT framework?\n\nLIMITATIONS:\nThe authors have addressed one of the limitations of the work, i.e. the assumption of known label distribution of test data. The authors have not addressed potential negative societal impacts of their work as it is mostly theoretical and methodological.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose Relative Entropic Optimal Transport, which allows to incorporate prior information matrix to the learning of optimal transport plan. After studying its theoritical properties, they adapt REOT to the long-tailed classification problem and establish the connection between optimal transport and classification. Finally, they illustrate the effectiveness of the proposed method via experiments on long-tailed classification and representation learning.\n\nSTRENGTHS:\nThe authors have well established the connection between REOT and various classification loss functions. They also clearly demonstrate how REOT can be used in long-tailed problem. The proposed method also shows strong performance in many long-tailed experiments.\n\nWEAKNESSES:\nI would say the contribution to the OT theory is quite limited and incremental because all theoritical results of REOT are straightforward adaptation from the EOT. The Prop 1 and its proof need serious revision because there are various typos, which leave me doubt on the correctness of the theoritical result and proof.\n\n- In Eq 20, 21, there is no consistent use of f and f', g and g'.\n\n- In Eq 21, there should be no epsilon. This makes the form of Qtilda incorrect because it depends on epsilon.\n\n- In Eq 22, 23, there is no epsilon.\n\n- In Eq 23, 24, there is no consistent use of f and f', g and g'.\n\n- Typos in Eq 25.\n\n- In Eq 25, I fail to see the last equality and to see why we can set f and g in such ways to get the equality. I feel like this is a circular reasoning.\n\n- In Prop 1, the authors state that when Qtilda = a otimes b, then the solution of REOT and EOT coincides (which is not true in general), while in the proof, they consider Qtilda = one matrix, which makes the two solution coincide.\n\n- There are also typos in Eq 42, 43. \n\nDespite some concerns on the theoritical result, I feel that it can be fixed without impacting the effectiveness and soundness of the proposed method.\n\nQUESTIONS:\n- I don't really understand the context of Figure 4: can the authors provide more description of the dataset and experiment?\n\n- Small remark: the authors might also want to add relevant reference to the barycentric projection:\n1) Ferradans, S.; Papadakis, N.; Peyré, G.; and Aujol, J.-F. 2014. Regularized Discrete Optimal Transport. SIAM Journal on Imaging Sciences, 7(3): 1853–1882.\n\n2) Courty, N.; Flamary, R.; Tuia, D.; and Rakotomamonjy, A. 2016. Optimal transport for domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39: 1853–1865.\n\nLIMITATIONS:\nThe authors do discuss the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

comments: SUMMARY:
This paper addresses the problem of unbalanced classification using a variant of optimal transport called Relative Entropic Optimal Transport (RE-OT), which alter the Sinkhorn distance with the prior information into the coupling solution. Experimental results across different domains validate the efficacy of their approach.

STRENGTHS:
The paper is easy to follow. The idea of using Barycentric projection used in previous work in domain adaptation with OT has been adapted  the context of (unbalanced) classification.

WEAKNESSES:
The proposed method much depends on the prior distribution Q which is supposed as the label distributions of the data. However, specifying correct Q is impossible in the reality. The author should show the effects of wrongly specified Q by experimental results. Otherwise, the proposed method has little applicable in the real world application.

QUESTIONS:
A recent work [1] has introduced a principled framework for using optimal transport for class-Imbalanced classification problem. Could you please differentiate the current work with this work in terms of theorical and experimental (if time allows) aspects. 
[1] Jin, Lianbao, Dayu Lang, and Na Lei. "An Optimal Transport View of Class-Imbalanced Visual Recognition." International Journal of Computer Vision (2023): 1-19.

LIMITATIONS:
The paper already mentioned its limitation: "It assumes known prior label distribution testing data, which is often unknown in real-world scenarios"
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis works proposed a new view to the classification problem through the lens of Optimal Transpot. They propose a new variant of OT, says Relative-OT, by changing the KL regularizer constraints in a given distribution. By some special properties of KL divergence, the works show a relationship betwen the entropic relative OT and the entropic OT in Proposition 1 and 2. \n\nIn the application of finding barycenter of multiple distributions, the authors proposed to iteratively change the relative distribution in the regularization to obtain a \"smooth\" (maybe meaningful) barycenter. \n\nAnother application is the long-tailed recognition problem. They define a cost matrix based on the logit value of the prediction neural network. Then they view the classification problem like a matching problem between data and labels. They aim to find the transport plan satisfying both constraints: one is from the true label, and one is from  the \"transport cost\" between data and probability value of a NN. For a long-tail issue, a relative distribution is set adaptively in the training process.  \n\nSection 5 shows the experiment results on LongTail data set and unbalanced data sets, like CIFAR10-LT, CIFAR100-, ImageNet_LT, molecule data etc.\n\nSTRENGTHS:\nI believe that the idea of using OT in the classification problem as presented is new. \n\nExperiment results show an improvement in performance compared to other methods but not in all cases.\n\nWEAKNESSES:\nI myself still do not get the justification for using the OT in classification problem. I mean all the use of relative distribution, iteration, restate the classification as two problems of finding optimal plan and distribution etc. I see some toy example of blur images for explanation, but apart from that I do not find any others, maybe I miss some parts of the paper. Here I mean the theory and intuition behind it, rather than just assemble OT and classification problem in the presented way. When more parameters involves in the training process of the model, it is more likely that we will obtain better performance.\n\nQUESTIONS:\n1. I do not understand the claim in equation (10). Could the author elaborate it?\n\n2. In Table 3, case of Many, the performance of proposed method is worse than some other methods. Could the authors provide explanations? \n\n\n\nMinors: typos Qij line 164, proposition 1: \"and are\"\n\nLIMITATIONS:\nIt is fine.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the relative entropic (RE) regularization for optimal transport (OT) problems. Instead of the traditional entropy regularizer (Cuturi'13), the RE-OT is defined through a given prior matrix $\\boldsymbol{Q}$, which guides the matching between source and target distributions. Using the approach of Inverse RE-OT, the authors derive a similar form to softmax-based cross-entropy loss to train long-tailed data in computer vision pipelines.\n\nSTRENGTHS:\n- The paper introduces RE-OT, a new variant of entropic regularization for OT. \n- The RE-OT enjoys some theoretical properties including a static Schrodinger form and a dual formulation as for the standard entropic regularizer. \n- The paper investigates the Inverse RE-OT for long-tailed recognition tasks and shows that these tasks can be formulated as matching perspectives with OT. \n- Extensive numerical experiments on image classification and molecule classification.\n\nWEAKNESSES:\n- The RE-OT is strongly dependent on the prior $\\boldsymbol{Q}$, that can be an additive hyperparameter.\n- The Inverse RE-OT approach is limited since one has to have a good knowledge of the supervision $\\tilde{\\boldsymbol{P}}.$\n\nQUESTIONS:\n### Questions:\n- Does the prior guide $\\boldsymbol{Q}$ belong, in general, to the polytope $U(a,b)$?\n- In the experiments, how the parameter $\\epsilon$ was chosen?\n- During training, the constant $c$ is it fixed? How can guarantee the positiveness of $\\boldsymbol{C}_{ij}$?\n- In L554, the last approximation is obtained by a Taylor expansion? \n\n### Typos\nBelow I list some typos to be corrected:\n- L38: involves learning the Inverse OT: add a reference\n- L105: $\\tilde{\\boldsymbol{P}}$ is not defined\n- L163:  bold $Q$\n- L164: $\\tilde{\\boldsymbol{Q}}_{ij}$\n- L165: ${\\boldsymbol{P}}^\\epsilon_{\\tilde{\\boldsymbol{Q}}}$\n- L173, L175: bold $K$\n- L180: in Eq (10), there is no transpose \n- L184: as $\\lambda$ changes.\n- L192 - L202: avoid to use $n$ as  the iterated index ($^{(n)}$)\n- L210: constant $c$ is not commented \n- L211: in Eq (14), the cost matrix $\\boldsymbol{C}$ depends in $\\theta$\n- L222: is an epoch\n- L240: in Eq (16) there is a dependency on $\\theta$. Also, a minus sign and a factor $n$ are missing, i.e. \n\\begin{equation}\n\\min_{\\theta} L = - \\sum_{i,j} \\tilde{\\boldsymbol{P}}_{ij} \\log \\frac{\\cdots} {n \\cdots}\n\\end{equation}\n- L258: deep graph matching: add a reference \n- L282, L283: croponding --> corresponding\n- L503: Eqs (22) and (23): missing dependency on $\\epsilon$\n- L507: multiplier --> multipliers\n- L507: $\\tilde{\\boldsymbol{Q}}_{ij}$\n- L508: in Eq (25), $\\boldsymbol{P}^\\epsilon_Q = \\cdots {\\boldsymbol{Q}} diag(e^{g'/\\epsilon})$\n- L510: \\boldsymbol{Q}\n- L522: $-\\boldsymbol{C}_{ij}$\n- L542 and L543: depedency on $\\theta$\n- L547: Eq (44) \n$$\n\\min_{\\theta} L = - \\sum_{i,j} \\tilde{\\boldsymbol{P}}_{ij} \\log \\frac{\\cdots} {n \\cdots}\n$$\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-08-02"}