Relative Entropic Optimal Transport: a (Prior-aware) Matching Perspective to (Unbalanced) Classification
Authors: Liangliang Shi
Haoyu Zhen
Gu Zhang
Junchi Yan
Venue: NeurIPS 2023 poster (2023)
Keywords: Optimal Transport; Unbalanced Classification

Abstract: Classification is a fundamental problem in machine learning, and considerable efforts have been recently devoted to the demanding long-tailed setting due to its prevalence in nature. Departure from the Bayesian framework, this paper rethinks classification from a matching perspective by studying the matching probability between samples and labels with optimal transport (OT) formulation. Specifically, we first propose a new variant of optimal transport, called Relative Entropic Optimal Transport (RE-OT), which guides the coupling solution to a known prior information matrix. We gives some theoretical results and their proof for RE-OT and surprisingly find RE-OT can help to deblur for barycenter images. Then we adopt inverse RE-OT for training long-tailed data and find that the loss derived from RE-OT has a similar form to Softmax-based cross-entropy loss, indicating a close connection between optimal transport and classification and the potential for transferring concepts between these two academic fields, such as barycentric projection in OT, which can map the labels back to the feature space. We further derive an epoch-varying RE-OT loss, and do the experiments on unbalanced image classification,  molecule classification, instance segmentation and representation learning. Experimental results show its effectiveness.

Relative Entropic Optimal Transport: a (Prior-aware) Matching Perspective to (Unbalanced) Classification
Liangliang Shi, Haoyu Zhen, Gu Zhang, Junchi Yan"
Dept. of Computer Science and Engineering & MoE Key Lab of AI, Shanghai Jiao Tong University
(shiliangliang, anye_zhen, blake-nash, yanjunchi)@sjtu edu.
PyTorch Code: ottt comm/Lianggliagggh//BBB.
Abstract
Classification is fundamental problem in machine learning, and considerable efforts have been recently devoted to the demanding long-tailed setting due to its prevalence in nature. Departure from the Bayesian framework, this paper rethinks classification from a matching perspective by studying the matching probability between samples and labels with optimal transport (OT) formulation. Specifically, we first propose new variant of optimal transport, called Relative Entropic Optimal Transport (RE-OT), which guides the coupling solution to a known prior information matrix. We gives some theoretical results and their proof for RE-OT and surprisingly find RE-OT can help to deblur for barycenter images. Then we adopt inverse RE-OT for training long-tailed data and find that the loss derived from RE-OT has similar form to Softmax- based cross-entropy loss, indicating a close connection between optimal transport and classification and the potential for transferring concepts between these two academic fields, such as barycentric projection in OT, which can map the labels back to the feature space. We further derive an epoch-varying RE-OT loss, and the experiments unbalanced image classification, molecule classification, instance segmentation and representation learning. Experimental results show its effectiveness.
1 Introduction
Long-tailed label distribution or unbalanced classification in training given more balanced testing set
has been a challenge for existing neural models 42
3 62 Various approaches have been developed,
such as Class Re- be-balancing 3 25
35|
g
45
52
Information Augmentation [36 77], and Module
Improvement 665 16 62.
However, the problem is still far from being solved.
Most existing methods on (long-tailed) classification study the conditional probability function p(ylæ:) their joint distribution given the label y and sample almost taking it for granted. For instance, [45] shows that in the long-tailed case, according to Bayes' theorem, the vanilla Softmax classification affected by the label distribution shift, which makes the classifier more inclined consider the samples as belonging to the majority class. To address this issue, the work explicitly considers the label distribution shift and re-derives the Softmax function. [22] argues that the conditional distribution ([y|z) is not the same between training and testing data, which can result in target-shift. To enhance uaaslannb learning. the authors propose to relax the assumption that the source and target domains share the same conditional distribution p(rly). Despite the de-facto Bayesian perspective for understanding classification, this paper, we take a matching view and demonstrate how new insights and approaches can be derived, especially for the challenging long-tail cases.
*Coreespnddence author. This work was partly supported by NSFC (62222607, 61972250, U19B2035) and Shanghai Municipal Science and Technology Major Project (2011SH7D77X0102)
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

From the matching perspective, the problem becomes matching samples to labels. During train- ing, the goal is learn the distances between the features of samples and labels. During testing, matching results can be obtained by calculating the distances between the features of testing samples and their labels. We formulate this matching using the theory of Optimal Transport (OT), where training precisely involves learning the Inverse OT [12 33 [1], and testing involves optimizing the transportation results. In the commonly studied long-tailed setting (35), the labels of training samples follow long-tail distribution, while the labels are uniformly distributed for testing data. With our OT formulation, we demonstrate that these two different priors (i.e. long-tailed and uniform) can effectively reduce the difference between training and testing, thereby increasing the accuracy of the matching probability.
To incorporate the prior information e.g. the aforementioned typical long-tailed setting, we introduce the amoothing-guider prior matrix Q to vanilla entropic OT and call the resulting formulation relative entropic OT (RE-OT). To this new variant of OT, we provide convergence analysis, its corresponding static Schrödinger form, Sinkhorn algorithm for solving the problem as well as the dual formulation of RE-OT, which are mostly similar to those for the vanilla entropic OT. We find our RE-OT can help to deblur for barycenter images by setting specific smoothing prior Q.
Specifically, we develop the inverse RE-OT (as our relative entropy OT's counterpart to inverse OT) to learn the cost matrix or equivalently, the sample features for training data. And then we use the RE-OT to optimize the transportation to get the matching results (i.e. predictions for testing data). It is uncovered that many coftmax-based cross-entropy losses (e.g. Balanced Softmax, A-softmax and etc) are special cases of our derived new loss under half constraints of OT by specifying different forms of the cost matrix, supervision matrix in IOT and the soooothhingggiieed prior matrix Q. In other words, we can generalize the cross-entropy loss by specifying different cost matrix, matrix, or other settings in the framework of inverse OT. We also find that when the regularization coefficient + 0, the loss of inverse RE-OT equals triplet loss in metric learning fields. Additionally, our matching perspective reveals interesting connections between OT and classification tasks. For example, we discover the equivalence between the coefficient of entropic regularization in OT and the temperature in Softmax. We also find that the concept of barycentric projection in OT can be used in classification to calculate the class barycenters in hidden feature space. We believe that the connection between these two fields will inspire further algorithmic development for both sides. This paper contributes in the following ways:
sooothinngguidder
1) We propose matching perspective to revisit and provide new understanding to the classification problem with OT formulation. The matching perspective provide a clear interpretation to the challenge of long-tailed classification, which lies in the inconsistency of the utilized prior information between the matches learned from the training set and those from the testing set.
2) We propose a new variant of entropic optimal transport, called relative entropic OT (RE-OT), for learning matching with specified prior. We provide theoretical results on various aspects of RE-OT, including its solution properties. Sinkhorn algorithm, barycenter calculation, and its dual form. As a side product of our study, we find that RE-OT can help to deblur barycenter images by setting specific smoothing prior Q, which entropic OT alone cannot achieve.
3) We further develop Inverse RE-OT to link our results to classification, and show that Softmax and its variants are special cases of our derived loss based on Inverse RE-OT. This finding inspires simple yet effective technique in practice for long-tailed classification: we let the smoothing-guider matrix Q to be spoch-varying. which gradually provides prior information and improves the performance.
4) Our study has revealed significant link between Optimal Transport (OT) and existing classification tasks, such as the relationship between the regularization coefficient in OT and the temperature in Softmax. This suggests that OT concepts can be applicd to classification tasks, and vice versa. For instance, the barycentric projection concept in OT can be utilized in classification tasks to calculate the class barycenters in the hidden feature space. We anticipate that this connection between the two fields will lead to further algorithmic development for both OT and classification.
5) Our method performs competitively and often outperforms baselines on various long-tailed bench- marks ranging from image classification, to molecule property prediction, and instance segmentation. Source code will be made publicly available.
2

---

2 Preliminaries and Related Work
Optimal Transport. Known as Wasserstein distance or Earth Mover's distance, Optimal transport (OT) [26). has gained significant attention in various fields including domain adaption M. generative models 15 34 49]. and image registration (14]. We begin by briefly introducing discrete Optimal Transport (OT), which provides a framework for viewing the (oog-tailed) classification problem from a matching perspective. For a more detailed introduction to OT, readers are directed to [43].
Consider two histograms E En and b E E.,. where the simplex En (x IR4 Jar 1, = 1}. We can represent the transportation polytope U(a, b), which the polyhedral set of m matrices:
(a b) = (P B+ P1m a. P 1, = b},
(1)
where 1,, and 1m and m dimensional vectors of ones. ((a b) contains all m nonnegative probabilisticc matrices with row and column sums a and b. Then the Kantorovich's optimal transport problem can be defined as 266
< C.P >= >= (a)(a
min
Ec,P.
where C E RT is the cost matrix for the distance between sample {x,}== and 1y The above minimization is a linear program and the optimal solution P can be obtained with the network simplex [54 or other off-th techniques e.g. 440, which require significant time overhead. For
its cost-effectiveness, the entropic regularization 57
is more popular and it gets approximation:
min (aU(a.b
L =<
C, P (H(P),
where the regularizer is specified as H(P) =
H Eij PPy(log(P.))) - 1). can be proved that
when c - 0, the unique solution of Eq. 3 converges to the optimal P of original problem. And when e + +00, we can get
(4) where @ b = a E U(a, b). So can find the entropic regularization (P) exactly do smoothing to the solution towards a @ b. Recently, Inverse Optimal Transport (IOT) start to be
P.
b,
studied 112 33 [1 50) which assumes that the cost C is unknown for learning, given the established coupling. The inverse of entropic OT problem can be formulated as:
min KL(P)P")
where
pd arg min PeU((.,)
<C
(5)
proposes a principled approach to infer the unknown costs and ( develops the mathematical theory for IOT. [33 shows that by IOT, one can also modify and improve the raw matching prediction by the cost, which is learned by the raw matching as noise supervision.
Specifically, 51
In this paper, we develop an IOT-based formulation to learn the cost between samples and their labels, which further allows one to learn the feature extractor of the training samples. Then with the features exacted from testing data, we match them with target labels and achieve the goal of classification.
Long-tailed recognition. Long-tailed recognition has been a standing problem with wide attention in
vision and learning [64 19 355. One common approach [45] 41 is to re-balance the class distribution in the training set. For example, some works employ re-sampling (25 or weighting (8 techniques to let the model pay more attention to the minority classes in the training set. Data augmentation is also adopted synthesize more samples for the underrepresented classes (28). While (9] transfer learning is used, which leverages knowledge learned from a source domain with sufficient data to improve the performance on target domain with imbalanced data. This line of research includes techniques such as fine-tuning (63] domain adaptation [53] and meta-learning [56). Recently, some works have proposed to explicitly model the long- tailed distribution, such as by adjusting the loss or designing specialized network architectures. Examples include focal loss [35], class- balanced loss (8), balanced Softmax loss 45). In this paper, we view classification from a matching perspective. Our method does not predict the label or class directly but learns the features in training and calculates the matching probability instead. With the aid of OT, we formulate with relative entropic as proposed in the next section, which adopts different prior for training and testing.
3

---

E= 0.01
6 0.1
E 1
10
Figure I: Results of P. by varying given the features (trained with SimCLR (5]) of ten CIFAR-I0 images and their augmentation. The cost matrix is set with the cosine distance between features. When - 0, QQ becomes sharper for probability prediction and when e - +o0, PQQ is more close Q. Here set Qıj 0.7 if sample and j are in the same class and Qij = 0.3 otherwise.
Relative Entropy Regularization for Optimal Transport
Motivation of introducing prior into OT. As introduced in Eq. entropic regularization (ER) is an effective method [IO] for approximating the Kantorovich OT solution. It pushes the original linear programming solution away from the hard boundary and obtains smoother solution by minimizing e e-strongly convex function. With the increase of the penalty coefficient, the solution progressively approaches a more 'smooth' solution, ultimately converging to b as illustrated by Eq. 4 However, question raised: such a smoothing direction a b appropriate for all practical situations, e-g. the long-tailed problems? Can we freely choose more suitable (smoothing) direction to achieve more effective results tailored to the problem or dataset at hand? this section, we will develop relative entropic OT (RE-OT), a more general formulation than traditional entropic OT 57
matrix C E R2xm and two margins E,, and b Σm,, where E,, and Σim represent the probability simplex with n and m bins, respectively, we introduce the positive senotthinggguidane matrix Q use the relative entropy regularizer HQ(P) in place of the traditional entropy in Eq, 3 to achieve this Ps. ability, defined as HQ(P) = = Cig Pij log Qu
Formulation.
Given
haauully-sppecffie
constant prior to guide the smoothing. We
1). Then we have:
min (eU(a.b
Ca C, -HHa(P).
(6)
The relative entropy HQ(P) can increase the probability of transportation for the element Pij if Qij larger. Consequently, if the prior in the form of Q available, the use relative entropy regularization can improve the efficiency of learning the transportation solution. Therefore, for longetailed tasks, we can assign a large Q element to the majority classes and a low element of Q to the tailed classes to enhance training. We will discuss this in detail in the next section.
For Eq. [6] we further study its properties. Firstly, we show the relation between RE-OT and the entropic OT by Eq. 3 as follows. The proof is given in Table
Proposition 1 (Solution Propriety.). Assume P P and are the optimal solution of RE-OT and entropic regularized OT, respectively. Then we can get:
(I) When e > +0c, the optimal RE-OT's solution P22 will converge to Q where Q takes the form
Q (aloglu)Qdiayyy with two uniquely defined non-negative vectors u and v.
(2) With the prior Q and its corresponding Q as defined in (1), we have P = P And when
b, have the equality P P
Proposition shows the relation between RE-OT and entropic OT, and
find RE-OT is
more general formulation. Note Q defined by and can be obtained by iteratively normalizing row/column sums of Q. Fig. [ shows the results of P by varying As increases, the optimal
coupling becomes more dense, but P becomes more uniform while Pa converges to Q
Besides, similar to the standard entropic regularized OT, our RE-OT in Eq 0 can also be reformulated the "static Schrödinger problem" form 31 which exactly leans projection under KL divergence.

---

Proposition 2 (static Schrödinger form). Redefine general KL divergence
Ps
KL(PK) P
log Kij
Pi Kij.
1/
The optimization in Eq.
6 is equivalent to the following minimization, where Kij Qize-Cisle Cislt
P2
-arggin KL(P (EU(a.b)
(8)
The proof is in Appendix B Prop. shows the optimal solution P22 is exactly the KL projection of K onto (a, b). In fact, K can be set K Qije -C,/ due the equivalence between P. and P Expect for the different form of K, RE-OT and entropic OT share the same static Schrödinger formulation, which means that many methods in entropic OT can applied to RE-OT, such as iterative Bregman projections (2)
The Sinkhorn algorithm and barycenters for RE-OT. The optimal solution for RE-OT can be estimated using the Sinkhorn algorithm. as its counterpart has already been well-developed for solving entropic OT [TO]. The algorithm for RE-OT shares the same form as that for entropic OT, given K; = Qize Csy/e, Specifically, with two non-negative vectors u' and v' uniquely defined up to multiplicative factor, the optimal solution has the form P. diag (u' [ diag(v'), which can be efficiently computed by iterating u', v' < a./Kv', b./K uu. The proof and algorithm are presented in Appendix C In addition, the barycenter between distributions is natural extension of OT 112 and with the matrix K, [2] defined it with a weighted KL projection problem:
min Σ CLLA, K)) s.t. P. 1, b,, Pilm = P21m ..... Pslm. = (9) (P-).a
where {A are the weights, b, are known histograms representing images, and a is the barycenter histogram to be calculated. Though casier to calculate compared with the regularized formulation in [1. drawback of this entropic-based method is that it can lead to blurred barycenters, and methods have been proposed to address this issue (23 By calculating barycenters between noise and an image (i.e. S = 2 here), we show that the blurred problem can be simply solved by setting À where
\1 X, 22 = and the prior matrix Q
Q (1- A)(P.) + +AP
(10)
where P is the optimal solution of entropic OT from image b to b2 Fig. 2 iillustrates the effect of different choices of Q when computing barycenters between noise and an image. Without using Q, the resulting barycenters are very blurry. When we set Q to be P the barycenters display the shape of the leopard clearly, but fail to transition smoothly from noise to leopard image as changes. To address this issue, we set Q as shown in Eq. I0 which can blur the image while transitioning smoothly with the gradient.
Proposition 3 (Dual formulation). From the optimization in Eq. 6 we can get its dual formulation:
L6 =< < -e f/c Kesle
(11)
where f E RD and g E mm are the corresponding dual variables.
The proof is given in Appendix D Exactly f, g are linked to u, v appearing in Sinkhorn algorithm by exp (f/c) and exp (g/ /), and thus Sinkhom algorithm can be done in log-domain [47]
Setting Q with the Optimal Solution Iteratively. The intuition for the prior is to iteratively update
the solution P as a new i.., (()) P Q where Q(t) represents the solution obtained after
the n-th iteration. The question then arises to how Q(!) changes over time. We find that this problem is equivalent to a proximal point algorithm:
( = arg min (
(12)
Here F(Q) =< C2Q Hle(,,)) Q), where (a)(a,)) (Q) is defined such that lor(a,)) (Q) -0 if
Q U(a, b), and (b)(a (a(aa))) (Q) too otherwise. We find this problem is discussed by 29 43
5

---

without Q
Varying Q= PP
==
== 1/5
2
= 3/5
2= 4/5
Figure 2: The barycenter results between noise and leopard image. It is very blurry without use of Q. However, when we set Q P the resulting barycenters clearly display the shape of the leopard, but fail transition smoothly from noise to leopard image as changes. To address this, we set Q as shown in Eq. 10 which blurs the image while transitioning with the increase of X.
which aims to get the optimal solution the non-regularized OT. As discussed Sinkhorn of RE-OT, have the optimal solution form:
Q (t) = diag (u Q (t-1) Oe C/e diag(v(t-1))
(13)
= diag(u (iag(n((-1) (1-1) ①
r++1c
diag(v (0) (..(((
When Q()) luxm, the optimization equivalent to applying Sinkhorn's 's algorithm itera- tively with a kernel e (1i
i.e. with decaying regularization coefficient f+I This observation
further emphasizes the importance of selecting an appropriate Q for RE-OT.
4 RE-OT for Long-tailed Recognition
We now discuss the application of inverse RE- OT to address the long-tailed classification task (including the contrastive learning setting). Un- like previous works that study this problem ing conditional probability, our approach aims learn features by matching samples with their labels using Inverse OT. Specifically, given the mini-batch pair set ((x;, y where x; is training sample and y; is the m mimimensional one-hot label of x,. We can the the cost Cg between sample x; and another one-hot la- bel given the neural network fo. Without loss of generality, we can set C = - lij where ly = (fe(x:)); is the j-th component of logits fo(x.) for sample x,. Then we can learn the matching between features and labels with inverse OT via a bi-level optimization:
Training
Coupling p?
Supervision
Matrix
Leoering Fnvense OI
label (e.g. one-hot)
Prior
feature extractor
Cost Matrix
Predic tion
Learning
Testing Data
Inference
Figure 3: RE-OT for unbalanced classification.
min KL(P|P") s.t. pa = =arg min < C., P -cHQ(P), 0
(14)
PEU
where P is the supervision given the label (y:}. For example, j-th component of y,. Here refer to the constraints for the coupling P. We can set U = Ü(a. b) with the full matching constraints its relaxation e.g. U (aa) (P)P1,.. a).
set P. = Yú, when yıj is the
For our designed smoothing guidance Q, simple idea is to set with the long-tailed ratio i.e. the training sample ratio of class labels which are readily available from the training which equals to

---

Table 1: Previous works in the view of inverse RE-OT by setting different ground truth P ijs cost Cij. prior Qiy and coefficient é under the constraints (a).
Formulation
Methods
Ground Truth P.,
Cost Ciyy
Q
penalty coefficient
Vanilla Sofimax
P., Mi
C₁ --
Q.
Focal
P.j Sy + P.,)
C2 ml-=e-fu by
Q,y =1
c=r Q.y =1
A-sofumax 17
P., =
Ciy e- [[.])v(0,..)
<
Tripkt 15
P. =Vo
C, [I)(())) - A(a))11
Qu
Claes-Ralaneel Loss
8
Pu (1 A1/ (1-
Co ------ fg
Qu =1
<m
LDAM Loss
P. =
=o=o-tu fy
Qu = e 11tm000
6=
Bolancol-Sofmmox 45
P,
C =-==-be
Qu My
Our Setting
P. Mh
C - by
Q (1 Mel) Uniform AAgle
the Balanced Softmax method proposed in [45). In this paper, motivated by Eq. Dolaving good efficiency in OT and other works [60] which adopt two stages for learning, we let it vary over epochs:
Q = (1 x()) Uniform + +X(t)r,
(15)
where t is the training epoch and r is the prior probability matrix e.g. the balanced ratio in teacher-basee prior predictions or the sample-class-wis setting discussed in Appendix E an epoch varying weight. When is small, x(t) is close to O, and Q more likely be uniform distribution, which gives the model less prior information. As approaches the final training epoch number T. X()) gradual process of introducing prior information can be helpful for training the inverse OT. We provide a specific setting for A(t) and r in Appendix E During the inference process, the problem reduces to the inner optimization in Eq. [ which takes the form of traditional unsupervised OT. We use this optimization to obtain the matching between the sample features and labels, which corresponds to classification for the testing data. We do not adopt the long-tailed setting of Q during inference because the testing data assumed to be uniform.
45
X is
and Q + r, which means the model is given the full prior information. This
4.1 Generalization to a Family of Losses in Form of Softmax-based Cross-Entropy
We find our new formulation with inverse OT is a more general form if we set U = ((a) where a, 1/n and many Softmax-based cross-entropy losses can be our special case by varying the form
of P, C and Q Specifically, with U = (a),, Eq 14 equals to
Que Ctylc
min B
Σ Pig log 2.3
(16)
Em Que Chale"
The proof is given in Appendix F cross-entropy loss. In particular, if we set P. C Q and in different forms as shown in Tab. | many classification loss can be the special cases of our formulation, as discussed in detail as follows.
We can find the form in Eq. 16 is similar with the Softmax
Special cases by varying C For matching problem, it is quite important to define a distance among samples. We first set CØ c c- byy where c a large enough value. In this case, Eq. reduces to normal softmax- basedd cross-entrops loss when Q is in the uniformly distributed. Alternatively, we can set C differently to suit specific tasks. For example, in learning in the hypersphere space, the Vanilla Softmax will be converted to A-softmax [37 to enforce angular margin constraints for improved face recognition performance.
Deriving more losses by varying Q. By varying the setting of Q, we can derive many Softmax- based losses as special cases of the RE-OT-based loss. For example, the Balanced Softmax can be obtained by setting Qij ni., where ny is the frequency of label j in the training set. This choice of Q encourages the model to assign equal weights each class during training. which helps to mitigate the impact of class imbalance.
Discussion on P. Re-weighting the example is a common way to improve the Long-tailed learning, including the empirical Re-weighting methods (e.g. focal loss (35 ) and automatic Re-weighting ways (e.g. 1,2RW [46)). In the view of IOT, the supervision the coupling P is self adapting and may depend on Pd which may be helpful in IOT application fields such deep graph matching.
7

---

Softmax
Focal
Ours
train
plane (head)
car (head)
bird (head)
test
horse (tail)
ship (tail)
truck (tail)
barycenter
Figure 4: t-SNE for training and testing sample features and class barycenters. The class barycenters are calculated by Eq. with features of training data. It can be observed that in the training data, the class barycenters are mainly on the feature centers of corresponding category samples, which proves the effectiveness of the barycentric projection map defined in Eq. [8] At the same time, we also find that barycenters calculated by features of training samples may not necessarily be centered in the test data. Additionally, the head class occupies significantly larger space than the tail class in the 2D feature space, which may be one the reasons for the poor accuracy the tail class.
Triplet loss for + 0+. Though Eq. 16 has very different form compared to triplet loss, the cquivalence can also be established when the temperature e + 0+. Note that the idea of the relation between triplet loss and our loss is motivated by the work (55) which gives the proof of the equivalence between triplet loss and InfoNCE. We give the proof in Appendix G
Representation Learning with our Formulation Our matching perspective can be applied not only to unbalanced classification but also to unbalanced representation learning, which we only need to modify the original one-hot label to the augmentation's features. In Sec. 5 conducted experiments on long-tailed contrastive learning and verified that by selecting an appropriate Q, we can improve the effectiveness of representation learning on long-tailed datasets.
4.2 Bridging Regularized OT to Classification
We have shown the classification can viewed as a matching problem and if we formulate the matching with optimal transport, can get the equivalence between Softmax (as well as its variants) and RE-OT under the setting of (a)) We can observe that there are some interconnected concepts between OT and classification, which can shed lights on deeper understanding of OT and in turn the classification problem with different forms. One notable example is the entropy regularization coefficient in OT and the temperature coefficient in Softmax. There have been numerous works [24 61 exploring the role of the temperature hyperparameter Softmax. We can understand it easier from the view of OT, where serves relaxation hyperparameter for couplings yielding smoother predictions. In fact, there are other concepts, such as barycentric projection in OT [3] which may help us understand more about classification. We give its definition here:
Definition 1 (Barycentric Projection). Consider the setting of Eq. 6 in which we use entropic regularization to approximate OT between discrete measures. One can define the so-called barycentric projection map with the coupling P
T T T Puu
(17)
where {ai) C. X is the set of locations corresponding to simplex a and {u;} is the set of locations in the Y space croponding simplex b.
From the classification view, exactly T(x:) is probability confidences of samples x; which can be understand the feature in the one-hot label space. Besides, motivated by this barycentric projection in OT, we can exactly define the mapping for the feature of one-hot label in hidden feature space:
1 T' U EY E.P.J
P.JREFF
(18)
where f; is the feature of x; extracting from the neural networks and F the feature space. Eq. maps the (one-hot) label to the feature space, which means need to learn the centroid features

---

as done in oneline clustering methods 4 32 but calculate it statically with Eq. 18 if the coupling known. As shown Fig. 4 we first train the models using vanilla softmax, focal loss, and our loss on the Cifar10-L.T dataset. We consider the logits as the sample features and save the logits from the three head and three tailed classes, along with the corresponding predicted couplings, as matrices. Then, we calculate the barycenters of the labels using Eq. [8] which effectively computes the weighted average of the features. We concat the logits and barycenters as a new matrix and use it to calculate t-SNE results for training data, as shown in the first row of Fig. 4 For testing data, we concat the testing logits with the barycenters calculated from the training data and use t-SNE dimensional reduction, as shown in the second row in Fig. 4 By comparing the first row and the second row of Fig. 4 we can observe the differences in the positions of the barycenters, which are actually caused by shift in the feature distributions.
Further Discussions between OT and classification. Note that many theoretical aspects and varia- tions of OT can be incorporated into the field of classification. Here are a few advantages unique to the OT perspective that can be achieved but are not possible in the traditional Bayesian view: 1) Variable constraints instead of {P- : P1 a). such as (modified) Optimal Partial Transport (OPT) with (P: P1 < <a, a, 1, P1 = s} where is the number requiring to predict. This allows for distributions that are not just one-vs vs-evsllll and enables rejection of classification the model determines that a sample cannot be classified into any of the candidate labels; 2) Generalization of softmax. The current softmax formulation is essentially based on Entropic Regularization of OT. However, OT regularization goes beyond just the entropic regularization and includes other regularizations such as L.2, Tsallis entropies, or baagennce-based ones. This opens up possibilities for generalizing softmax; 3) Classification cross-representatioon models. From the OT perspective, classification between the features of samples and labels, based on different feature spaces. For example, the sample features may be in 100 dimensions while the label features are in 20 dimensions. This can be helpful for preserving privacy as Gromov- Wasserstein Distance [44] only requires similarity between samples and labels. Due to the various variants and theories within OT, considering classification as OT naturally allows us to incorporate existing OT knowledge into the field of classification, which is the ongoing work we are conducting.
Distance, can be performed in
Experiments
We evaluate our proposed approach for both image and molecule datasets. As the primary focus of this paper is the impact of loss functions on model learning, we do not adopt additional specialized techniques to improve performance, such as transfer learning or resampling. This is to control variables and thus all the baselines used in this study represent only the loss functions proposed by them. Additional experimental settings and details can be found in Appendix
Experiments on long-tailed image classification. The long-tailed image classification datasets analyzed in this study include CIFARIO-LT. CIFARI00-LT [30], and Imagenet-LT (38). To evaluate the performance of the models, we use the corresponding balanced test dataset and report top-1 accuracy. Specifically, for CIFAR-10, we report accuracy on two sets of classes: Many-shot (more than 100 images) and Few-shot (less than 100 images). For CIFAR- 100 and Imagenet, we report accuracy on three sets: Many-shot (more than 100 images), Medium-shot (20 100 images), and Fhw-shot (less than 20 images). The experiments for unbalanced image classification are all conducted with an imbalanced factor of less than 200, which is defined as the ratio of the number of training instances in the largest class to the smallest (45. The results for long-tailed classification are presented in Table Note that in our approach, the teacher-based method involves setting the prior matrix in Eq. 15 using a vanilla softmax trained teacher model. On the other hand, the ratio-based method refers to setting r as the long-tailed ratio. From a comprehensive perspective, at least one of approaches achieves the best average accuracy across the entire dataset.
Experiments on unbalanced molecule classification. We further examine the application of our proposed method in the context of molecular representation learning task, which holds significant implications for healthcare [59] and drug discovery 58). Specifically, we perform the experiments on two unbalanced molecule classification datasets, i.e. DGBG-MOLBBBR and
from the Open Graph Benchmark (OGB) [2I]. We use the default train/val/test split with ratio 8:1:1. Given the presence of class imbalance within the two datasets, we have chosen to adhere to the precedent set by prior studies, also employing the ROC-AUC as the metric for performance evaluation. As the experimental results are casily influenced by initialization, conducted five repetitions of

---

Table 3: Top-1 accuracy (%) for long-tailed image classification with 200 imbalanced factor.
CIFARIO-LT
CIFARIO-LLT
ImageNet-I.7
Method
Many
Few
All
Many
Medium
Few
All
Many
Medium
Few
All
Vanilla Softmax
77.4
68.9
74.9
75.8
48.2
11.0
42.0
57.3
26.2
3.1
35.0
Focal Loss
35
79.6
58.4
73.3
76.1
46.9
11.1
41.7
57.3
27.6
4,4
35.9
LDAM 3
80.5
65.2
75.9
75.7
50.6
11.5
42.9
$7.3
27.6
35.9
LogitAdjust
80.0
75.7
4.1
36.5
14.0
CB-CE 8
76.6
70.7
74.8
53.2
48.8
13.3
36.3
35.3
32.1
21.2
31,9
CB-FC 8
76.6
70.7
74.8
53.2
48.8
13.3
36.3
35.3
32.1
21.2
31.9
Balanced Soflmax
35
82.2
71.6
79.0
70.3
50.4
26.5
47.0
52.5
38.6
17.8
41.1
Murs(techer-baase
84.4
63.7
78.2
69.7
55.6
24.9
47.9
41.5
37.6
31.1
38.2
Qurs(ratio- based)
81.5
74.6
79.4
70.4
53.0
26.6
47.9
53.5
39.0
17.4
41.6
the experiment and reported the mean accuracy and its standard deviation in the Tab. the effectiveness of our method.
which show
Experiments Long-
Table 2: Test on instance segmentation and molecule classification.
Tailed Instance Segmen- tation. We do experi- ments on long-tailed in- stance segmentation with LVIS v1.0 datasets (18), as one of the most chal- lenging datasets in vision with much higher imbal- ance factor compared to the rest. We use the offi- cial splits and evaluation conducted on validation set. The setting mainly follows the experiments in 445, whose details given in Appendix H
Method
LVIS (instunce seg.)
OGBG-MOLBBII
OGBC-MOLBACE
APm
AP;
AP.
AP,
ROC-AUC
ROC-AUC
Vanilla
Focal Loss
55
19.69
8.75
17.74
26.68
68.15 + 1.29
80.65 + 1,50
3
7.14
13:23
20.89
79.20 +
LogitAdjust 39
22.41
11.70
21.18
28.48
69.05
1.59
81.25 1: 0.33
CR-CF
20.38
9.58
18:72
26.98
69.19 1.18
79.39 + 1.16
25.25
80.24
Balanced Softmax
22.60
12.88
21.20
28.44
68.13 - 0.87
80.26 1: 2.28
Oursfratio-haseel
APm. AP,, ., AP AP, are reported in Tab. [ AP,, denotes Average Precision masks, AP,, AP.. AP, denote Average Precision of masks rare classes, common classes and frequent classes. As shown in Tab. 2] we can find that our method achieves the best APm on the entire set and perform competitively for other evaluations.
More Experiments. We also perform experiments on Long-Tailed Tailed contrastive representation learning on CIFAR I0-IT, CIFAR100-LT with 100 imbalanced factor. We compare our methods with the methods in 27 The settings and results are given in Appendix(H.S) Besides, to further demonstrate the usefulness of viewing classification OT, new inference method is proposed for testing. replaced the softmax (i.e. constraints within U(a) with Sinkhorn algorithm (i.e. constraints within (a, b)), where b represents the assumed ratio in the testing data (e.g., long-tailed, uniform, or reverse toiget distribution). Details are given Appendix H.6
6 Conclusion
We have provided a matching perspective to provide new understanding to classification. Under our matching-based framework, we show that the inherent challenge of long-tailed classification lies in the inconsistency of the prior information used for training and testing under the matching formulation. We then develop inverse relative entropic OT approach to revisit the classification problem, and especially for the long-tailed case, develop simple yet effective technique based on our theoretical insights. Experimental results verify the effectiveness of our methods.
Broader impacts. Our work establishes a theoretical framework for understanding and solving sspervisue problems using OT theory. This not only broadens the range of applications for OT in generative models, such as WGAN and its variants, but also provides a new perspective for redefining, understanding, and solving concepts of OT can facilitate the discovery of interesting relationships and insights for supervised problems, such as the connection between open set recognition and unbalanced OT. Limitations. assumes known prior label distribution testing data, which is often unknown in real-world scenarios.
problems. Furthermore, the theory and
10

---

References
[1] M. Agueh and G. Carlier. Barycenters in the wasserstein space. SIAM Journal on Mathematical Analysis, 43(2):904-99) 924, 2011.
[2] J.-D. Benamou, G. Carlier, M. Cuturi, L. Nenna, and G. Peyré. Iterative bregman projections for regularized transportation problems. SIAM Journal on Scientific Computing, 37(2):11111- A1138, 2015.
[3] K. Cao, C. Wei, A. Gaidon, N. Arechiga, and T. Ma. Learning imbalanced datasets with siseel-dstriutiin-awee margin loss. Advances in neural information processing systems, 32, 2019.
[4] M. Caron, I. Misra, Mairal, Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-999 9924, 2020.
[5] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607, PMLR, 2020.
[6] W.- Chiu, P. Wang, and P. Shafto. Discrete probabilistic inverse optimal transport. In International Conference on Machine Learning, pages 3925-3946. PMLR, 2022.
[7] J. Cui. Z. Zhong, S. Liu, B. Yu, and J. Jia. Parametric contrastive learning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 715-724, 2021.
[8] Y. Cui, M. Jia, T.-Y. Lin, Y. Song, and S. Belongie. Class-balanced loss based effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2019.
[9] Y. Cui, Y. Song. Sun, A. Howard, and S. Belongie. Large scale fine- grained categorization and domain-specifti transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4109-41 4118, 2018
[10] M. Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
[11] B. B. Damodaran, B. Kellenberger, R. Flamary, D. Tuia, and N. Courty. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In Proceedings the European conference on computer vision (ECCV), pages 447-463, 2018.
[12] A. Dupuy, A. Galichon, and Y. Sun. Estimating matching affinity matrix under low-rank constraints. arXiv preprint arXiv:1612.0958; 2016.
[13] S. Ferradans, N. Papadakis. G. Peyré. and J.-F. Aujol. Regularized discrete optimal transport. SIAM Journal on Imaging Sciences, 7(3): (133)1853-882 2014.
[14] J. Feydy, B. Charlier, F.-X Vialard, and G. Peyré. Optimal transport for diffeomorphic reg- istration. In Medical Image Computing and Computer Assisted Intervention- MICCAI 2017: 20th International Conference, Quebec City, QC, Canada, September 11-13, 2017. Proceedings. Part I 20, pages 291-299. Springer, 2017.
[15] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville, Improved training of wasserstein gans. In NIPS, 2017.
[16] H. Guo and S. Wang. Long-tailes multi-label visual recognition by collaborative training on uniform and re-balanced samplings. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15098, 2021.
[17] Z. Guo, W. Yu, C. Zhang, M. Jiang, and N. V. Chawla. Graseq: graph and sequence fusion learning for molecular property prediction. In Proceedings of the 29th ACM international conference information & knowledge management, pages 435-443, 2020.

---

[18] A. Gupta, P. Dollar, and R. Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5364, 2019.
[19] H. He and E. A. Garcia. Learning from imbalanced data. IEEE Transactions on knowledge and data engineering, 21(9):1233-1288) 2009.
[20] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[21] W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and Leskovec. Open graph benchmark: Datasets for machine learning graphs. Advances in neural information processing systems, 33:221 18 221333., 2020.
[22] M. A. Jamal, M. Brown, M.-H. Yang. L. Wang, and B. Gong. Rethinking class-balanced methods for long-tailed visual recognition from domain adaptation perspective. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7610-7619, 2020.
[23] H. Janati, M. Cuturi, and Gramfort. Debiased sinkhorn barycenters. International Conference Machine Leurning, pages 4692-4701. PMLR, 2020.
[24] E. Jang, S. Gu, and B. Poole. Categorical reparameterizatioo with gumbel-softmax. arXiv preprint arXiv:1611.01 2016.
[25] B. Kang, S. Xie, M. Rohrbach, Z. Yan, A. Gordo, J. Feng, and Y. Kalantidis. Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv: 2019.
[26] L. Kantorovich. On the transfer of masses (in russian). In Doklady Akademii Nauk, volume 37, pages 227-229, 1942.
[27] P. Khosla, P. Teterwak, C. Wang. A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33: 18:16661-18673 2020.
[28] Kim, Jeong, and Shin. M2m: Imbalanced classification via major-to- to-minor translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13896-13900 13905, 2020.
[29] J. J. Kosowsky and A. L. Yuille. The invisible hand algorithm: Solving the assignment problem with statistical physics. Neural networks, 7(3):477-490, 1994.
[30] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[31] C. Léonard. From the schrödinger problem to the monge-kantorovich problem. Journal of Functional Analysis, 262((): 179-1990, 2012.
[32] J. Li, P. Zhou, C. Xiong, and S. C. Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint arXiv:2005.04900 2020.
[33] R. Li. X. Ye, H. Zhou, and H. Zha. Learning to match via inverse optimal transport. Journal of machine learning research, 20, 2019.
[34] Y. Li, Y. Mo, L. Shi, and J. Yan. Improving generative adversarial networks via adversarial learning in latent space. Advances in Neural Information Processing Systems, 35:8868-888 8881, 2022.
[35] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[36] J. Liu, Y. Sun, C. Han, Z. Dou, and W. Li. Deep representation learning on long-tailed data: A learnable embedding augmentation perspective. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 2970-2977 2020.
12

---

[37] W. Liu, Y. Wen, Z. Yu, M. Li, B. Raj, and L. Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 212-220, 2017.
[38] Z. Liu, Z. Miao, X. Zhan, J. Wang, B. Gong, and S. X. Yu. Large-scale long-tailed recognition an open world. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2537-2546, 2019.
[39] A. K. Menon, S. Jayasumana, A. S. Rawat, H. Jain, A. Veit, and S. Kumar. Long-tail learning via logit adjustment. arXiv preprint KKIv:20077003114 2020.
[40] J. Orlin. A faster strongly polynomial minimum cost flow algorithm. Proceedings of the Twentieth anmual ACM symposium Theory of Computing, pages 377-387, 1988.
[41] S. Park, J. Lim, Y. Jeon, and J. Y. Choi. Inftuence- loss for imbalanced visual classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 735-744, 2021.
[42] H. Peng, M. Sun, and Li. Optimal transport for long-tailed recognition with learnable cost matrix. In International Conference on Learning Representations, 2022.
[43] G. Peyre and M. Cuturi. Computational optimal transport. Foundations and Trends in Machine Learning, 115-66)355-667, 2019.
[44] G. Peyré, M. Cuturi, and J. Solomon. Gromov-wasserstinn averaging of kernel and distance matrices. In International conference on machine learning, pages 2664-2672. PMLR, 2016.
[45] J. Ren, C. Yu, X. Ma, H. Zhao, S. Yi, et al. Balanced meta-softmax for long-tailed visual recognition. Advances in neural information processing systems, 33:4175-4186, 2020.
[46] M. Ren, W. Zeng, Yang, and R. Urtasun. Learning to reweight examples for robust deep learning. In International conference on machine learning, pages 4334 4343. PMLR, 2018.
[47] B. Schmitzer. Stabilized sparse scaling algorithms for entropy regularized transport problems. SIAM Journal on Scientific Computing, 41(3)A1443-A148) 2019.
[48] Schroff, D. Kalenichenko, and Philbin. Facenet: unified embedding for face recogni- tion and clustering. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 813.823, 2015.
[49] L. Shi, Y. Li, and J. Yan. lid-gan: an iid sampling perspective for regularizing mode collapse. arXiv preprint arXiv:2106. 000000000063.... 2021.
[50] L. Shi, G. Zhang, H. Zhen, J. Fan, and J. Yan. Understanding and generalizing contrastive learning from the inverse optimal transport perspective. International conference on machine learning, 2023.
[51] A. M. Stuart and M.-T. Wolfram. Inverse optimal transport. SIAM Journal on Applied Mathe- matics, 80(1):5996619 2020.
[52] J. Tan, C. Wang, B. Li, Q Li, W. Ouyang, C. Yin, and J. Yan. Equalization loss for long-tailed object recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11662-16671, 2020.
[53] E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 7167-7176, 2017.
[54] G. R. Waissi. Network flows: Theory, algorithms, and applications, 1994.
[55] F. Wang and H. Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2495-25952 2504,, 2021.
[56] Y.-. Wang, Ramanan, and M. Hebert. Learning to model the tail. Advances neural information processing systems, 30, 2017.
13

---

[57] A. G. Wilson. The use of entropy maximising models, in the theory of trip distribution, mode split and route split. Journal of transport economics and policy, pages 108-122 126, 1969.
[58] N. Yang, K. Zeng, Q. Wu, X. Jia, and Yan. Learning substructure invariance for out-of- distribution molecular representations, In Advances in Neural Information Processing Systems, 2022.
[59] N. Yang, K. Zeng, Q. Wu, and J. Yan. Molerec: Combinatorial drug recommendation with substructure-avwe molecular representation learning. In Proceedings of the ACM Web Confer- ence 2023, pages 4075-4085, 2023.
[60] S. Zhang, Z. Li, S. Yan. X. He, and J. Sun. Distribution alignment: A unified framework for long-tail visual recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2361-286 2370,, 2021.
[61] X. Zhang, F. X. Yu, S. Karaman, W. Zhang, and S.-F. Chang. Heated-up softmax embedding. arXiv preprint arXiv: /809004157 2018.
[62] Y. Zhang, B. Hooi, L. Hong, and J. Feng. Self-supervised aggregation of diverse experts for test-agnostic tailet recognition. arXiv preprint arXiv:2107.09249 2021.
[63] Y. Zhang, B. Hooi, D. Hu, Liang, and J. Feng. Unleashing the power of contrastive self- supervised visual models via contrast-regularizee fine-tuning. Advances in Neural Information Processing Systems, $4:29848-29980 2021.
[64] Y. Zhang, Kang, B. Hooi, Yan, and J. Feng. Deep long-tailed learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[65] B. Zhou, Q Cui, X.-S. Wei, and Z.-M. Chen. Bbn: Bilateral Bilateral-branch bbrnch network with cumulative learning for long-tailed visual recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9719-9728, 2020.
14


Reviews:
[{"review_id": "xZW6qmy87I", "reviewer": "Reviewer_Cbtk", "summary": "The paper introduces the relative entropic (RE) regularization for optimal transport (OT) problems. Instead of the traditional entropy regularizer (Cuturi'13), the RE-OT is defined through a given prior matrix $\\boldsymbol{Q}$, which guides the matching between source and target distributions. Using the approach of Inverse RE-OT, the authors derive a similar form to softmax-based cross-entropy loss to train long-tailed data in computer vision pipelines. ", "questions": "### Questions:\n- Does the prior guide $\\boldsymbol{Q}$ belong, in general, to the polytope $U(a,b)$?\n- In the experiments, how the parameter $\\epsilon$ was chosen?\n- During training, the constant $c$ is it fixed? How can guarantee the positiveness of $\\boldsymbol{C}_{ij}$?\n- In L554, the last approximation is obtained by a Taylor expansion? \n\n### Typos\nBelow I list some typos to be corrected:\n- L38: involves learning the Inverse OT: add a reference\n- L105: $\\tilde{\\boldsymbol{P}}$ is not defined\n- L163:  bold $Q$\n- L164: $\\tilde{\\boldsymbol{Q}}_{ij}$\n- L165: ${\\boldsymbol{P}}^\\epsilon_{\\tilde{\\boldsymbol{Q}}}$\n- L173, L175: bold $K$\n- L180: in Eq (10), there is no transpose \n- L184: as $\\lambda$ changes.\n- L192 - L202: avoid to use $n$ as  the iterated index ($^{(n)}$)\n- L210: constant $c$ is not commented \n- L211: in Eq (14), the cost matrix $\\boldsymbol{C}$ depends in $\\theta$\n- L222: is an epoch\n- L240: in Eq (16) there is a dependency on $\\theta$. Also, a minus sign and a factor $n$ are missing, i.e. \n\\begin{equation}\n\\min_{\\theta} L = - \\sum_{i,j} \\tilde{\\boldsymbol{P}}_{ij} \\log \\frac{\\cdots} {n \\cdots}\n\\end{equation}\n- L258: deep graph matching: add a reference \n- L282, L283: croponding --> corresponding\n- L503: Eqs (22) and (23): missing dependency on $\\epsilon$\n- L507: multiplier --> multipliers\n- L507: $\\tilde{\\boldsymbol{Q}}_{ij}$\n- L508: in Eq (25), $\\boldsymbol{P}^\\epsilon_Q = \\cdots {\\boldsymbol{Q}} diag(e^{g'/\\epsilon})$\n- L510: \\boldsymbol{Q}\n- L522: $-\\boldsymbol{C}_{ij}$\n- L542 and L543: depedency on $\\theta$\n- L547: Eq (44) \n$$\n\\min_{\\theta} L = - \\sum_{i,j} \\tilde{\\boldsymbol{P}}_{ij} \\log \\frac{\\cdots} {n \\cdots}\n$$", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper introduces RE-OT, a new variant of entropic regularization for OT. \n- The RE-OT enjoys some theoretical properties including a static Schrodinger form and a dual formulation as for the standard entropic regularizer. \n- The paper investigates the Inverse RE-OT for long-tailed recognition tasks and shows that these tasks can be formulated as matching perspectives with OT. \n- Extensive numerical experiments on image classification and molecule classification.", "weaknesses": "- The RE-OT is strongly dependent on the prior $\\boldsymbol{Q}$, that can be an additive hyperparameter.\n- The Inverse RE-OT approach is limited since one has to have a good knowledge of the supervision $\\tilde{\\boldsymbol{P}}.$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable time and comments and we hope the following answers can address your concerns.\n\n>***Q1: Does the prior guide Q belong, in general, to the polytope U(a,b)***\n\nA1: No. For example, $Q$ can be set as $1_{n×m}\\notin U(a,b)$, and then the RE-OT problem degenerates into Entropic OT. However, the resulting $\\tilde{Q}$ computed based on $Q$ belongs to the polytope $U(a,b)$. As demonstrated in Prop.1 (2), the use of $Q$ and $\\tilde{Q}$ is essentially equivalent.\n\n>***Q2: In the experiments, how the parameter $\\epsilon$ was chosen?***\n\nA2：We simply set $\\epsilon=1$. Here $\\epsilon$ is exactly equivalent to the temperature $\\tau$ in the Softmax function. In our paper, we did not modify the temperature and followed the setting of Vanilla Softmax, where we simply set $\\epsilon=1$.\n\n>***Q3: During training, the constant c is it fixed? How can guarantee the positiveness of $C_{ij}$***\n\nA3:  $c$ is fixed but is independent of training.  Here, $c$ refers to a sufficiently large number that exists only in theory. We can easily observe that $C_{ij}=c-l_{ij}>0$ when $c$ is sufficiently large, ensuring the positivity of $C_{ij}$. However, in practical computations, when performing row normalization, we can obtain $e^{-(c-l_{ij})}/\\sum_k e^{-(c-l_{ik})}=e^{l_{ij}}/\\sum_k e^{l_{ik}}$. The results are independent of $c$.\n\n>***Q4: In L554, the last approximation is obtained by a Taylor expansion?***\n\nA4: Yes, the last approximation is obtained through Taylor expansion as $\\epsilon$ approaches 0 from the positive side. This derivation is inspired by Eq. 6 in [1]. We will clarify it in the new version.\n\n>***Q5: The Inverse RE-OT approach is limited since one has to have a good knowledge of the supervision $\\tilde{P}$***\n\nA5: Yes, one has to know $\\tilde{P}$ for Inverse RE-OT; otherwise, we cannot optimize Inverse RE-OT for representation learning. However, for a classification task, $\\tilde{P}$ is assumed to be known as the labels of samples.\n\n>***Q6:The RE-OT is strongly dependent on the prior Q, that can be an additive hyperparameter.***\n\nA6: Yes, RE-OT is dependent on the prior $Q$ which can be an additive hyperparameter. However, we try to address your concern about the application of OT and (long-tailed) classification fields:\n\n(1) For OT, $Q$ can be **computed** based on practical needs, instead of artificially adding a hyperparameter. For instance, as shown in Eq. 10, by setting $Q$ in a specific way, we can deblur the barycenter of images, which is not achievable with traditional Entropic OT.\n\n(2) For (long-tailed) classification, the optimal $Q$ is indeed unknown, and the known label ratios serve as a surrogate choice. However, we believe that one day, we will be able to discover a more suitable $Q$ to enhance representation learning. In fact, we are currently adopting a new $Q$, which is computed using a vanilla softmax-trained \"teacher\" model. We present the results in the uploaded PDF in Tab. 3 and it is interesting to find a great improvement for the head classes, which inspires us to further explore the choice of $Q$.\n\n>***Q7: About Typos.***\n\nA7: Thank you for your feedback. We will make corrections in the final version.\n\n[1]  F. Wang and H. Liu. Understanding the behavior of contrastive loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2495–2504, 2021\n"}, {"review_id": "Ur1711PpEV", "reviewer": "Reviewer_RCtv", "summary": "This works proposed a new view to the classification problem through the lens of Optimal Transpot. They propose a new variant of OT, says Relative-OT, by changing the KL regularizer constraints in a given distribution. By some special properties of KL divergence, the works show a relationship betwen the entropic relative OT and the entropic OT in Proposition 1 and 2. \n\nIn the application of finding barycenter of multiple distributions, the authors proposed to iteratively change the relative distribution in the regularization to obtain a \"smooth\" (maybe meaningful) barycenter. \n\nAnother application is the long-tailed recognition problem. They define a cost matrix based on the logit value of the prediction neural network. Then they view the classification problem like a matching problem between data and labels. They aim to find the transport plan satisfying both constraints: one is from the true label, and one is from  the \"transport cost\" between data and probability value of a NN. For a long-tail issue, a relative distribution is set adaptively in the training process.  \n\nSection 5 shows the experiment results on LongTail data set and unbalanced data sets, like CIFAR10-LT, CIFAR100-, ImageNet_LT, molecule data etc. ", "questions": "1. I do not understand the claim in equation (10). Could the author elaborate it?\n\n2. In Table 3, case of Many, the performance of proposed method is worse than some other methods. Could the authors provide explanations? \n\n\n\nMinors: typos Qij line 164, proposition 1: \"and are\" ", "limitations": "It is fine.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I believe that the idea of using OT in the classification problem as presented is new. \n\nExperiment results show an improvement in performance compared to other methods but not in all cases.", "weaknesses": "I myself still do not get the justification for using the OT in classification problem. I mean all the use of relative distribution, iteration, restate the classification as two problems of finding optimal plan and distribution etc. I see some toy example of blur images for explanation, but apart from that I do not find any others, maybe I miss some parts of the paper. Here I mean the theory and intuition behind it, rather than just assemble OT and classification problem in the presented way. When more parameters involves in the training process of the model, it is more likely that we will obtain better performance. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable time in reviewing this paper. Here are our responses.\n\n>***Q1: \"I do not understand the claim in equation (10). Could the author elaborate it?\"\"***\n\nA1: Certainly. Here is our explanation. Assume that $P^\\epsilon$ is a coupling from image $a$ to image $b$, and we can obtain the normalized pixel values $b_j=\\sum_i P^{\\epsilon}_{ij}$.  \n\nWhen $b_j$ is small (indicating a white region at position $j$), the corresponding entries $P^{\\epsilon}_{:,j} $  will also be small. \n\nConsequently, when computing the barycenter $c$, by setting $Q=P^{\\epsilon}$ as the coupling from $a$ to $c$, the solution $P_{:,j}$ will be influenced by $Q$, resulting in a small value for $c_j$. This leads to a non-blurry image. Considering the transportation from $b$ to $a$, we interpolate between $P^{\\epsilon}$ and its transpose to obtain the final coupling matrix $Q$.\n\n>***Q2: In Table 3, case of Many, the performance of proposed method is worse than some other methods. Could the authors provide explanations?***\n\nA2: This scenario is quite common also for many peer methods in long-tailed experiments. Essentially, it represents a tradeoff between the head and tailed samples. To achieve higher accuracy for the tailed classes, there is a certain sacrifice in accuracy for the head class. However, the accuracy of the whole data tends to improve as a result.\n\n>***Q3: \"I myself still do not get the justification for using the OT in classification problem.\"***\n\nA3: We think that viewing classification as OT is highly valuable. Although we mentioned in the future work section that the OT perspective might help approach open-set problems, we can elaborate more on this direction if you are willing to hear about it. Here are a few directions we can do from the OT perspective yet less possible in the traditional Bayesian way:\n\n**1.  Change in the coupling constraint  $\\\\{ P: P1=a \\\\}$**.  We can adopt other constraints instead of $\\\\{ P: P1=a \\\\}$, such as using the constraints in (modified) Optimal Partial Transport (OPT) with $\\\\{ P: P1 \\leq a,1^\\top P1=s \\\\}$. This allows the model that is not just choosing one from all classes and enables rejection of classification if the model determines that a sample cannot be classified into any of the candidate labels.\n\n**2. Change in label selection in one model**. In the OT perspective, classification is not understood in terms of $P(y|x)$ but rather from a matching perspective. Therefore, when we have representations of samples and labels, we can freely choose different label sets to form different classification tasks. For example, during training, we classify into ${y_1,y_2,\\dots,y_{10}}$, but during testing, we classify into ${y_3,y_7,y_{10}}$. Additionally, if the label representation is based on NLP embedding models, we can include previously unseen labels in the classification testing phase, potentially addressing zero-shot learning problems without the need for additional parameter training.\n\n**3. Generalization of Softmax using other regularization.** The current softmax formulation is essentially based on the Entropic Regularization of OT. However, OT regularization goes beyond just the entropic regularization and includes other regularizations such as L2, Tsallis entropies, or divergence-based ones. This opens up possibilities for generalizing softmax.\n\n**4. Classification cross-representation models using Gromov-Wasserstein Distance.** From the OT perspective, classification between the features of samples and labels, based on the Gromov-Wasserstein Distance, can be performed in different feature spaces. For example, the sample features may be in 100 dimensions while the label features are in 20 dimensions. This can be helpful for preserving privacy as Gromov-Wasserstein Distance only requires similarity between samples and labels.\n\nDue to the various variants and theories within OT, considering classification as OT naturally allows us to incorporate existing OT knowledge into the field of classification, which is the ongoing work we are conducting. Only through this OT perspective, it becomes possible in the future to develop models that can \"Classify Anything,\" including different task selections, variable classification constraints, and more in one model. Therefore, the main purpose of this paper is to provide a different perspective on classification models and offer conceptual assistance to general recognition models.\n\n>***Q4: About Typos.***\n\nA4: Thank you for your feedback. We will make corrections in the final version.\n\n"}, {"review_id": "qEzSyRpBk9", "reviewer": "Reviewer_pKsm", "summary": "This paper addresses the problem of unbalanced classification using a variant of optimal transport called Relative Entropic Optimal Transport (RE-OT), which alter the Sinkhorn distance with the prior information into the coupling solution. Experimental results across different domains validate the efficacy of their approach.", "questions": "A recent work [1] has introduced a principled framework for using optimal transport for class-Imbalanced classification problem. Could you please differentiate the current work with this work in terms of theorical and experimental (if time allows) aspects. \n[1] Jin, Lianbao, Dayu Lang, and Na Lei. \"An Optimal Transport View of Class-Imbalanced Visual Recognition.\" International Journal of Computer Vision (2023): 1-19.\n", "limitations": "The paper already mentioned its limitation: \"It assumes known prior label distribution testing data, which is often unknown in real-world scenarios\"", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is easy to follow. The idea of using Barycentric projection used in previous work in domain adaptation with OT has been adapted  the context of (unbalanced) classification.", "weaknesses": "The proposed method much depends on the prior distribution Q which is supposed as the label distributions of the data. However, specifying correct Q is impossible in the reality. The author should show the effects of wrongly specified Q by experimental results. Otherwise, the proposed method has little applicable in the real world application.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time to review this paper. Below are our responses.\n\n>***Q1: Difference to [1]***\n\nA1: [1] views neural networks as mappings in optimal transport and proposes that these mappings need to adapt due to the difference between the training and testing label distributions. In terms of technical details, it primarily investigates the dual form of Kantorovich OT and establishes a connection between the gradient of the Kantorovich functional and cross-entropy. Modifying the Kantorovich functional for the inconsistency between the training and testing label distributions can improve testing predictions. In contrast, our paper shows the following differences:\n\n1. We do not start from the dual form but directly focus on the primary OT formulation, leading to the conclusion that Softmax is a special case.\n2. The theoretical approximation proposed in [1] relates gradients between cross-entropy and the Kantorovich functional, whereas our equivalence lies between Softmax and Kantorovich coupling, which is the main difference.\n3. Our paper is not limited to long-tailed problems alone. The main purpose of Section 4 is to illustrate that modified Softmax-CrossEntropy or other classification losses, from the OT  perspective, can be viewed as different OT settings. Consequently, we can conclude that classification can be understood as OT. In future works, we can then incorporate other theoretical advancements of OT into classification tasks. We provide some examples of using the OT perspective to improve classification in the response to Reviewer Cbtk in A3.\n\n>***Q2: About the $Q$ in real world.***\n\nA2: Yes, RE-OT is dependent on the prior $Q$ which may not exist in real-world. Here we give our ideas that may address your concerns:\n\n1. The optimal $Q$ is indeed unknown, and the known label ratios serve as a surrogate choice for (long-tailed) classification. However, we believe that one day, one will be able to discover a more suitable $Q$ to enhance representation learning. In fact, we are currently adopting a new $Q$, which is computed using a vanilla softmax-trained \"teacher\" model. We present the results in the uploaded PDF in Tab. 3 and it is interesting to find a great improvement for the head classes, which inspires us to further explore the choice of $Q$.\n\n2. The purpose of Sec.4 is mainly to give the perspective that the classification is OT. Many variants of OT can be introduced to the classification problem with this OT perspective (e.g. optimal partial transport with the constrain $\\\\{\\mathbf{P}\\mathbf{1}\\leq \\mathbf{a}, \\mathbf{1}^\\top \\mathbf{P}\\mathbf{1}=s\\\\}$. See more examples in A3 of the rebuttal for reviewer RCtv."}, {"review_id": "60NmtAna9q", "reviewer": "Reviewer_46Rh", "summary": "The authors propose Relative Entropic Optimal Transport, which allows to incorporate prior information matrix to the learning of optimal transport plan. After studying its theoritical properties, they adapt REOT to the long-tailed classification problem and establish the connection between optimal transport and classification. Finally, they illustrate the effectiveness of the proposed method via experiments on long-tailed classification and representation learning.", "questions": "- I don't really understand the context of Figure 4: can the authors provide more description of the dataset and experiment?\n\n- Small remark: the authors might also want to add relevant reference to the barycentric projection:\n1) Ferradans, S.; Papadakis, N.; Peyré, G.; and Aujol, J.-F. 2014. Regularized Discrete Optimal Transport. SIAM Journal on Imaging Sciences, 7(3): 1853–1882.\n\n2) Courty, N.; Flamary, R.; Tuia, D.; and Rakotomamonjy, A. 2016. Optimal transport for domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39: 1853–1865.\n\n", "limitations": "The authors do discuss the limitations of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors have well established the connection between REOT and various classification loss functions. They also clearly demonstrate how REOT can be used in long-tailed problem. The proposed method also shows strong performance in many long-tailed experiments.", "weaknesses": "I would say the contribution to the OT theory is quite limited and incremental because all theoritical results of REOT are straightforward adaptation from the EOT. The Prop 1 and its proof need serious revision because there are various typos, which leave me doubt on the correctness of the theoritical result and proof.\n\n- In Eq 20, 21, there is no consistent use of f and f', g and g'.\n\n- In Eq 21, there should be no epsilon. This makes the form of Qtilda incorrect because it depends on epsilon.\n\n- In Eq 22, 23, there is no epsilon.\n\n- In Eq 23, 24, there is no consistent use of f and f', g and g'.\n\n- Typos in Eq 25.\n\n- In Eq 25, I fail to see the last equality and to see why we can set f and g in such ways to get the equality. I feel like this is a circular reasoning.\n\n- In Prop 1, the authors state that when Qtilda = a otimes b, then the solution of REOT and EOT coincides (which is not true in general), while in the proof, they consider Qtilda = one matrix, which makes the two solution coincide.\n\n- There are also typos in Eq 42, 43. \n\nDespite some concerns on the theoritical result, I feel that it can be fixed without impacting the effectiveness and soundness of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for spending your valuable time on this paper. We hope that our responses can address the concerns you may have.\n\n>***Q1: \"In Prop 1, the authors state that when Qtilda = a otimes b, then the solution of REOT and EOT coincides (which is not true in general), while in the proof, they consider Qtilda = one matrix, which makes the two solution coincide.\"***\n\nA1: Thank you for the comments. We apologize for the misunderstanding. We have corrected the errors in the proof of Prop. 1, and you can see the updated version in the uploaded PDF if you are interested.  Here are our responses to your points:\n\n1. $\\tilde{Q}$ belongs to $U(a,b)$ due to the optimization problem $\\tilde{Q}=\\arg\\min_P -H_Q(P), s.t. P\\in U(a,b)$. \n\n$\\quad$ So, one can set $Q=\\mathbf{1}_{n\\times m}$ or ${Q}=a\\otimes b$. \n\n$\\quad$  But it is wrong to set $\\tilde{Q}=\\mathbf{1}_{n\\times m}$. \n\n$\\quad$  Specifically, when $Q=\\mathbf{1}_{n\\times m}$, we can simply obtain its corresponding $\\tilde{Q}=a\\otimes b$.\n\n2. Most papers adopt $Q=\\mathbf{1}_{n\\times m}$ for the entropic formulation. However, a few papers do use the form ${Q}=a\\otimes b$. Please see [1]. These two formulations are exactly equivalent, and the purpose of Prop.1 is to demonstrate that different $Q$ can have equal RE-OT solutions when they have the same $\\tilde{Q}$. \n\n$\\quad$  We have provided simple numerical experiments **in Fig. 2** in the uploaded PDF to show the **equivalence of $P_{Q}$ and $P_{\\tilde{Q}}$** . \n\n3. Sorry for the typos in the proof that may cause misunderstandings. We make corrections to the proof of Prop. 1 **in the uploaded PDF (Fig. 1)**.\n\n[1] Statistical bounds for entropic optimal transport: sample complexity and the central limit theorem, NeurIPS 2019.\n\n>***Q2: \"I don't really understand the context of Figure 4: can the authors provide more description of the dataset and experiment?\" ***\n\nA2: Thank you. The following is our description of the dataset and experiment of Fig.4. Exactly, in the experiment of Fig.4, we first train the models using vanilla softmax, focal loss, and our loss on the Cifar10-LT dataset. We consider the logits as the sample features (in theory, we could also choose intermediate features of the network instead) and save the logits from the three head and three tailed classes, along with the corresponding predicted couplings, as matrices.\n\nThen, we calculate the barycenters of the labels using Eq.18, which effectively computes the weighted average of the features. We concat the logits and barycenters as a new matrix and use it to calculate t-SNE results for the training data, as shown in the first row of Fig.4. For the testing data, we concat the testing logits with the barycenters calculated from the training data and use t-SNE dimensional reduction, as shown in the second row in Fig.4. By comparing the first row and the second row of Fig.4, we can observe the differences in the positions of the barycenters, which are actually caused by shift in the feature distributions.\n\n>***Q3: About typos and remark about adding relevant reference to the barycentric projection***\n\nA3：Thanks again. We will correct our mistakes and add barycentric projection references.\n"}, {"review_id": "SIRTXDGt9J", "reviewer": "Reviewer_HboX", "summary": "The paper proposes an inverse Relative Entropic Optimal Transport (RE-OT) point of view for classification problems. The paper then proposes to use inverse RE-OT with a time-varying prior for solving long-tailed classification problems. Evaluations show improvements compared to existing baselines such as vanilla softmax and balanced softmax on long-tailed tasks. ", "questions": "- How does (15) relate to (10)? \n- Are (17) and (18) unique to the OT framework?", "limitations": "The authors have addressed one of the limitations of the work, i.e. the assumption of known label distribution of test data. The authors have not addressed potential negative societal impacts of their work as it is mostly theoretical and methodological. ", "rating": 5, "confidence": 2, "soundness": 4, "presentation": 2, "contribution": 2, "strengths": "The reviewer believes the main strengths of the paper are solid theory and novelty of the framework. Specifically, \n\n- The OT and novel RE-OT theory is introduced clearly, with provided visualizations and explanations to aid understanding. \n- The novel RE-OT theory makes a nice connection and a unified view for many existing classification methods, as summarized in Table 1.\n- Experiment results show some advantage compared to existing methods. \n", "weaknesses": "The reviewer mainly has concerns with the impact of the work. In particular, \n\n- Some additional intuitions would be helpful for the reviewer on why RE-OT gives a good or useful way for tackling classification problems. Currently, the RE-OT framework is laid out clearly, and the authors proposed to apply RE-OT to solve long-tailed classification, but the paper does not make clear what methodological or practical benefit the proposed framework has. \n- Comparing different methods in Table 1, the newly proposed method only differs from Balanced-Softmax in the choice of Q slightly, and the proposed choice of Q is not properly motivated in my opinion. The work also does not compare with using epoch-varying losses or two-stage approaches for other methods. \n- The evaluations show relatively minor improvements compared to existing works, and most experiments do not report error bars so it is unclear if most improvements are significant. \n\nThe work also has some minor issues with clarity:\n- The work uses $n$ for both number of iterations and dimension of polytope $\\Sigma_n$ which can be confusing. \n- Proposition 3 appears to be misplaced. \n- The notations in Figure 2 and equation (10) are quite hard to understand. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the time and effort you have spent reviewing this paper. Below are our responses:\n\n>***Q1: the paper does not make clear what methodological or practical benefit the proposed framework has.***\n\nWe think that our (RE-)OT framework for classification is highly motivated and valuable. Although we mentioned in the future work section that the OT perspective might help approach open-set problems, we can elaborate more on this direction if you are willing to hear about it. Here are a few directions we can do from the OT perspective yet less possible in the traditional Bayesian way:\n\n**1.  Change in the coupling constraint  $\\\\{ P: P1=a \\\\}$**.  We can adopt other constraints instead of $\\\\{ P: P1=a \\\\}$, such as using the constraints in (modified) Optimal Partial Transport (OPT) with $\\\\{ P: P1 \\leq a,1^\\top P1=s \\\\}$. This allows the model that is not just choosing one from all classes and enables rejection of classification if the model determines that a sample cannot be classified into any of the candidate labels.\n\n**2. Change in label selection in one model**.  In the OT perspective, classification is not understood in terms of $P(y|x)$ but rather from a matching perspective. Therefore, when we have representations of samples and labels, we can freely choose different label sets to form different classification tasks. For example, during training, we classify into ${y_1,y_2,\\dots,y_{10}}$, but during testing, we classify into ${y_3,y_7,y_{10}}$. Additionally, if the label representation is based on NLP embedding models, we can include previously unseen labels in the classification testing phase, potentially addressing zero-shot learning problems without the need for additional parameter training.\n\nYou can see more in the A3 response of Reviewer RCtv (due to the length limitation).\n\nBecause of the various variants and theories within OT, considering classification as OT naturally allows us to incorporate existing OT knowledge into the field of classification, which is the ongoing work we are conducting. Only through this OT perspective, it becomes possible in the future to develop models that can \"Classify Anything,\" including different task selections, variable classification constraints, and more in one model. Therefore, the main purpose of this paper is to provide a different perspective on classification models and offer conceptual assistance to general recognition models.\n\n>***Q2: \"Comparing different methods in Table 1, the newly proposed method only differs from Balanced-Softmax in the choice of Q slightly, and the proposed choice of Q is not properly motivated in my opinion. The work also does not compare with using epoch-varying losses or two-stage approaches for other methods.\"***\n\nA2: Yes, we agree that our proposed choice of Q is not properly motivated and the newly proposed method only differs from Balanced-Softmax in the choice of $Q$ slightly though our main purpose is to show the conclusion that classification is OT. However, However, we believe that one day, one will be able to discover a more suitable to enhance representation learning. In fact, we are currently adopting a new $Q$, which is computed using a vanilla softmax-trained \"teacher\" model. We present the results in the uploaded PDF in Tab. 3 and it is interesting to find a great improvement for the head classes, which inspires us to further explore the choice of $Q$.\n\n>***Q3: \"The evaluations show relatively minor improvements compared to existing works, and most experiments do not report error bars so it is unclear if most improvements are significant.\"***\n\nA3: Thanks. We train the different models 10 times on CIFAR100 and the results are in Tab. 2 in the PDF file.\n\n>***Q4:How does (15) relate to (10)?***\n\nA4: The following are our thoughts that relate Eq.15 to Eq.10.  \n\nEq.10 provides a deblurred image by gradually optimizing Q, indicating that Q does not necessarily need to be set as a fixed matrix. Building upon the observation that a two-stage approach, where vanilla softmax is trained first and then the modified loss is trained, can yield good results, it is worth noting that directly training with the modified loss (e.g., Balanced softmax without vanilla Softmax pretraining) may lead to poorer performance. This suggests that different training stages require varying degrees of correction. Therefore, we propose Eq.15 as our loss function.\n\n>***Q5:Are (17) and (18) unique to the OT framework?***\n\nA5: To the best of our knowledge, yes, we have only encountered relevant definitions and formulas within OT. In fact, Eq.17 investigates the transformation of features between two different spaces based on coupling. It may not be immediately apparent to draw a direct connection between this problem with the traditional Bayesian perspective, which primarily focuses on studying the conditional probability $P(y|x)$.\n\n>***Q6:The work uses n for both number of iterations and dimension of polytope $\\Sigma_n$ which can be confusing.***\n\nA6: Thanks for pointing it out. We will correct it in the new version.\n\n>***Q7:Proposition 3 appears to be misplaced.***\n\nA7: Prop. 3 mainly follows the dual form of Entropic OT. Please see Proposition 4.4 in [1] (Page 77 if in the same version)\n\n[1] G. Peyre and M. Cuturi. Computational optimal transport. Foundations and Trends in Machine Learning, 11(5-6):355–607, 2019.\n\n>***Q8:The notations in Figure 2 and equation (10) are quite hard to understand.***\n\nA8: Figure 2 talks about the barycenter calculated by Eq. 9 between noise and a leopard image. The first row is the results based on Entropic OT, where the barycenter images are blurred. The following two rows are RE-OT-based results setting different $Q$. We will polish the image and description. Thanks.\n\nFor Eq. 10, we have interpreted in the response of A1 for the Reviewer RCtv. Due to the limited length of rebuttal, we don't repeat our answers. Thank you for your understanding.\n\n\n"}]

Author rebuttal:
We sincerely appreciate the reviewers for investing their valuable time and providing insightful comments on our paper. Overall, the reviewers found our work to be novel (Cbtk, RCtv), with promising experimental results (Cbtk, RCtv, HboX), and easy to follow (pKsm). Additionally, they acknowledged the theoretical properties (Cbtk), the meaningful barycenter calculation (RCtv), and the connections between REOT and various classification losses (46Rh, HboX). We are grateful for the recognition and encouragement from the reviewers, and their comments have inspired us to further improve our work.

Simultaneously, the reviewers have provided several critical insights and improvement suggestions. In response to these, we have carefully considered their feedback and provided our response. For instance, one common concern raised by multiple reviewers is the practical significance of viewing classification as OT. We have addressed this concern by offering explanations and providing several examples to illustrate the interesting and meaningful aspects of this perspective. We firmly believe that our work will bring further inspiration to the readers.



Meta-review: The reviewers found the paper interesting, with new entropic OT and inverse OT formulation and with surprising relations with (unbalanced) classification but had a few concerns such as practicality of the method, lack of details for numerical experiment, and writing, which lead to a borderline positive score.

The authors did a detailed rebuttal and provided additional experiments. The practicality concerns were discussed by the authors and they provided a new experiment with a teacher model. The missing references have been discussed well. The rebuttal was appreciated by the reviewers (one increased their score). 

The final decision is to accept the paper but to strongly suggest that the authors take into account the comments from the reviewers, integrate to the paper the new results and discussion and do a major proof checking of the paper since that need was noted by most reviewers.
Decision: Accept (poster)