CSOT: Curriculum and Structure-Aware Optimal Transport for Learning with Noisy Labels
Authors: Wanxing Chang
Ye Shi
Jingya Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Learning with Noisy Labels
Optimal Transport
Curriculum Learning
TL;DR: In this paper, we proposed Curriculum and Structure-aware Optimal Transport , a novel solution to construct robust denoising and relabeling allocator for learning with noisy labels.

Abstract: Learning with noisy labels (LNL) poses a significant challenge in training a well-generalized model while avoiding overfitting to corrupted labels.
Recent advances have achieved impressive performance by identifying clean labels and correcting corrupted labels for training.
However, the current approaches rely heavily on the model’s predictions and evaluate each sample independently without considering either the global or local structure of the sample distribution.
These limitations typically result in a suboptimal solution for the identification and correction processes, which eventually leads to models overfitting to incorrect labels.
In this paper, we propose a novel optimal transport (OT) formulation, called Curriculum and Structure-aware Optimal Transport (CSOT). CSOT concurrently considers the inter- and intra-distribution structure of the samples to construct a robust denoising and relabeling allocator.
During the training process, the allocator incrementally assigns reliable labels to a fraction of the samples with the highest confidence. 
These labels have both global discriminability and local coherence.
Notably, CSOT is a new OT formulation with a nonconvex objective function and curriculum constraints, so it is not directly compatible with classical OT solvers. 
Here, we develop a lightspeed computational method that involves a scaling iteration within a generalized conditional gradient framework to solve CSOT efficiently.
Extensive experiments demonstrate the superiority of our method over the current state-of-the-arts in LNL.

CSOT: Curriculum and Structure-Aware Optimal Transport for Learning with Noisy Labels
Wanxing Chang
Ye Shi
Jingya Wang 1.n
TehanghaiTech University
Shanghai Engincering Research Center of Intelligent Vision and Imaging
{changux, shiye, wangj (aaery))@shnnghatteeee edu. cn
Abstract
Learning with noisy labels (LNL.) poses a significant challenge in training a well- generalized model while avoiding overfitting to corrupted labels. Recent advances have achieved impressive performance by identifying clean labels and correcting corrupted labels for training. However, the current approaches rely heavily on the models predictions and evaluate each sample independently without consid- ering either the global or local structure of the sample distribution. These lim- itations typically result in suboptimal solution for the identification and cor- rection processes, which eventually leads models overfitting incorrect la- bels. In this paper, propose novel optimal transport (OT) formulation, called Curriculum and Structure-aware Optimal Transport (CSOT). CSOT concurrently considers the inter- and intra-distribution structure of the samples to construct a robust denoising and relabeling allocator. During the training process, the allo- cator incrementally assigns reliable labels highest confidence. These labels have both global discriminability and local co- herence. Notably, CSOT is new OT formulation with nonconvex objective function and curriculum constraints, so is not directly compatible with clas- sical OT solvers. Here, develop lightspeed computational method that in- volves a scaling iteration within a generalized conditional gradient framework solve CSOT efficiently. Extensive experiments demonstrate the superiority our method over the current tfate-of-the-arts in LNI.. Code is available at bttpp://giithue com Changusx/(SOOT-or--11 http://hhhb..ooo/aaaaaaa/////00/
fraction of the samples with the
Introduction
Deep neural networks (DNNs) have significantly boosted performance in various computer vision tasks, including image classification [133], object detection [[]]], and semantic segmentation [32]] However, the remarkable performance of deep learning algorithms heavily relies large-scale high- quality human annotations, which are extremely expensive and time-consuue consuming to obtain. Alterna- tively, mining large-scale labeled data based cost-effective way to collect labels, but this approach inevitably introduces noisy labels. Since DNNs easily overfit noisy labels Z71,, such label noise can significantly degrade performance,
web search and user tags [49, [ can provide
giving rise to a challenging task: learning with noisy labels (LNL) 500 52 [6].
Numerous strategies have been proposed to mitigate the negative impact of noisy labels, including loss correction based on transition matrix estimation [55,, re-weighting [60]] label correction [76]]
"Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

and sample selection [52]] Recent advances have achieved impressive performance by identifying clean labels and conrecting corrupted labels for training. However, current approaches rely heavily on the models predictions to identify or correct labels even if the model is not yet sufficiently trained. Moreover, these approaches often evaluate each sample independently, disregarding the global or local structure of the sample distribution. Hence, the identification and correction process results in a suboptimal solution which eventually leads to a model overfitting to incorrect labels.
In light of the limitations of distribution modeling, optimal transport (OT) offers a promising solution by optimizing the global distribution matching problem that searches for an efficient transport plan from one distribution to another. To date, OT has been applied in various machine learning tasks [ILI 83, 28]. In particular, OT-based yseudo-labeling [ILT] [33] attempts to map samples class centroids, while considering the inter-distriuutir matching of samples and classes. However, such an approach could also produce assignments that overlook the inherent coherence structure of the sample distribution, i.e. intra-disti distribttnitno coherence. More specifically, the cost matrix in OT relies on pairwise metries, so two nearby samples could be mapped to two far-away class centroids (Fig. m...
In this paper, to enhance intra-distribution coherence, we propose new OT formulation for denois- ing and relabeling, called Structure-aware Optimal Transport (SOT). This formulation fully consid- ers the intra- distribution structure of the samples and produces robust assignments with both global discriminability and local coherence. Technically speaking, we introduce local coherent regularized terms to encourage both prediction- and label-level local consistency in the assignments. Further- more, to avoid generating incorrect labels in the early stages of training or cases with high noise ratios, we devise Curriculum and Structure-aware Optimal Transport (CSOT) based on SOT. CSOT constructs a robust denoising and relabeling allocator by relaxing one of the equality constraints to allow only a fraction of the samples with the highest confidence to be selected. These samples are then assigned with reliable pseudo labels. The allocator progressively selects and relabels batches of high-confidence samples based on an increasing budget factor that controls the number of selected samples. Notably, CSOT is a new OT formulation with a nonconvex objective function and curricu- lum constraints, so it is significantly different from the classical OT formulations. Hence, to solve CSOT efficiently, we developed a lightspeed computational method that involves scaling iteration within a generalized conditional gradient framework [59].
Our contribution can be summarized as follows: 1) We tackle the denoising and relabeling problem in LNL from new perspective, i.e. simultaneously considering the inter- and intra-distribution structure for generating superior pseudo labels using optimal transport. 2) To fully consider the in- trinsic coherence structure of sample distribution, we propose a novel optimal transport formulation, namely Curriculum and Structure-aware Optimal Transport (CSOT), which constructs a robust de- noising and relabeling allocator that mitigates error accumulation. This allocator selects a fraction of high-confidence samples, which are then assigned reliable labels with both global discriminabil- ity and local coherence. 3) We further develop a lightspeed computational method that involves a scaling iteration within a generalized conditional gradient framework to efficiently solve CSOT. 4) Extensive experiments demonstrate the superiority of our method over state-of-the-art methods in LNL.
2 Related Work
Learning with noisy labels. LNL is a well-studied field with numerous strategies having been pro- posed to solve this challenging problem, such as robust loss design [82,, 20], loss correction [35, (56). loss re-weighting [60. XQ] and sample selection [522 31, [T]. Currently, the methods that are deliy- ering superior performance mainly involve learning from both selected clean labels and relabeled corrupted labels [96. gSJ. The mainstream approaches for identifying clean labels typically rely on the small-loss criterion [GT] 177, /ZI, [E]] These methods often model per-sample loss distributions us- ing a Beta Mixture Model [[ST]] or a Gaussian Mixture Model [52]], treating samples with smaller loss as clean ones [E, ZTL [6. The label correction methods, such as PENCIL [[6]], Selfie [[33], ELR [[50] and DivideMix [G) typically adopt pseudo- strategy that leverages the DNNs predictions to correct the labels. However, these approaches evaluate each sample independently without con- sidering the correlations among samples, which leads to a suboptimal identification and correction solution. To this end, some work [155, 45) attempt to leverage k-nearest neighbor predictions [6] for clean identification and label correction. Besides, to further select and correct noisy labels robustly.

---

OT Cleaner [73], as well as concurrent OT-Filter [23]] designed to consider the global sample distri- bution by formulating pseudo-labeling as an optimal transport problem. In this paper, we propose CSOT to construct a robust denoising and relabeling allocator that simultaneously considers both the global and local structure of sample distribution so as to generate better pseudo labels.
Optimal transport-basee pseudo-labeling. OT is a constrained optimization problem that aims to find the optimal coupling matrix to map one probability distribution to another while minimiz- ing the total cost [ OT has been formulated as pseudo-labeling (PL) technique for range of machine learning tasks, including Blass-imbalancee learning [ 28. 68], semi-supervised learn- ing [651, 54,, [.]], clustering [ES, I. E5J, domain adaptation [83, 121, label refinery [83, 68, DAX 23).
and others. Unlike prediction-base based PI. [62], OT-based PI. optimizes the mapping samples to class centroids, while considering the global structure of the sample distribution in terms of marginal con- straints instead of per-sample predictions. For example, Self-labelling [S] and SWAV [TT], which are designed for self-supervised learning, both seek optimal equal-partition clustering to avoid the models collapse. addition, because OT-based PL considers marginal constraints, it can also con- sider class distribution to solve Mlassimbblanne problems [ 28. 68] However, these approaches only consider the inter-distribution matching of samples and classes but do not consider the intra- distribution coherence structure of samples. By contrast, our proposed CSOT considers both the inter- and intra- dittadiitribution structure and generates superior pscudo labels for noise-robust learning.
Curriculum learning. Curriculum learning (CL) attempts to gradually increase the difficulty of the training samples, allowing the model to learn progressively from easier concepts to more com- plex ones [[2]. CL has been applied to various machine learning tasks, including image classifica- tion [3% SeI and reinforcement learning [53, 21. Recently, the combination of curriculum learning has become popular in semi-supervised learning. These methods mainly focus on dynamic confident thresholding [69, 29, [ZS] instead of adopting fixed threshold [62]]. Flex-
and pseudo-labe
match [78] designs class-wise thresholds and lowers the thresholds for classes that more difficult to learn. Different from dynamic thresholding approaches, SI.A [65]] only assigns pseudo labels to easy samples gradually based on an OT-like problem. In the context of LNI., CurriculumNet [30] designs a curriculum by ranking the complexity of the data using its distribution density in feature space. Alternatively, RoCL [KS] selects easier samples considering both the dynamics of the per-sample loss and the output consistency. Our proposed CSOT constructs a robust denois- ing and relabeling allocator that gradually assigns high-quality labels fraction of the samples with the highest confidence. This encourages both global discriminability and local coherence in assignments.
3 Preliminaries
Optimal transport. Here we briefly recap the well-known formulation of OT. Given two proba- bility simplex vectors a and B3 indicating two distributions, as well as a cost matrix C E R|aXx(8] where |a]] denotes the dimension of ox, OT aims seek the optimal coupling matrix Q by minimiz- ing the following objective
min (C,Qu
QEIT(a.
where (. denote Frobenius dot-product. The coupling matrix Q satisfies the polytope II( 33) 1 al = = where a and / are essentially marginal probability vectors. Intuitively speaking, these two marginal probability vectors can be interpreted as coupling budgets, which control the mapping intensity of each row and column in Q
{o e TR |aaxx @ Q1 ||||
T1
Q
Pseudo-labeling based on optimal transport. Let P E R BBC denote classifier softmax predic- tions, where B the batch size of samples, and C is the number of classes. The OT-based PL considers mapping samples to class centroids and the cost matrix C can be formulated as log P [65, 68). We can rewrite the objective for OT-based PI. based on Problem (II) as follows
min QEn # Ln. 1c)
(- log P, Q)
(2)
where 1a indicates d-dimensional vector of ones. The pseudo-labeling matrix can be obtained by normalization: BQ. Unlike prediction-based PL [2 which evaluates each sample independently,
3

---

Classical OT
Structure- OT (Ours)
Local Prediction Consistency
A
Labal Consiseacyy
Clean sumples
Corupttd samples
Imglleit class centroids
Selected and labeled
Decision boundary
30%
50%
m 100%
Figure 1: (Top) Comparison between classical OT and our proposed Structure-aware OT. Clas- sical OT tends mismatch two nearby samples to far-away class centroids when the decision boundary not accurate enough. To mitigate this, our SOT generates local consensus assignments for each sample by preserving prediction-level and label-level consistency. Notably, for vague ples located near the ambiguous decision boundary. SOT rectifies their assignments based on the neighborhood majority consistency. (Bottom) The illustration of curriculum denoising and relabeling based on proposed CSOT. The decision boundary refers to the surface that separates two classes by the classifier. The m represents the curriculum budget that controls the number of selected samples and progressively increases during the training process.
OT-based PL considers -nttr-iistribution matching of samples and classes, as well as the global structure of sample distribution, thanks to the equality constraints.
Sinkhorn algorithm for classical optimal transport problem. Directly optimizing the exact OT problem would be time-consuming, and entropic regularization term is introduced [[I)] mingem)a, (C, Q) + (Q, log Q) where = > 0. The entropic regularization term enables OT to be approximated efficiently by the Sinkhorn algorithm [IT), which involves matrix scaling iterations executed efficiently by matrix multiplication on GPU.
Methodology
Problem setup. Let Derain [xx,, Mi) J denote the noisy training set, where is an image with its associated label y/: over classes, but whether the given label accurate or not unknown. We call the correctly-labeles ones as cleun, and the mislabeled ones as corrupted. LNL aims train a network that is robust to corrupted labels and achieves high accuracy on a clean test set.
4.1 Structure-Aware Optimal Transport for Denoising and Relabeling
Even though existing OT-based PL. considers the global structure of sample distribution, the intrinsic coherence structure of the samples is ignored. Specifically, the cost matrix in OT relies on pairwise metrics and thus two nearby samples could be mapped to two far-away class centroids. To further consider the intrinsic coherence structure, we propose Structure-aware Optimal Transport (SOT) for denoising and relabeling, which promotes local consensus assignment by encouraging prediction- level and label-level consistency, as shown in Fig.
Our proposed SOT for denoising and relabeling is formulated by adding two local coherent regular- ized terms based Problem (2). Given cosine similarity S E
among samples in feature space, a one-hot label matrix L E BBC transformed from given noisy labels, and a softmax pre-
diction matrix E R
BxC SOT is formulated as follows
min QEn fe
(- log P. Q) (Ω (Q) +Q)(())
(3)

---

where the local coherent regularized terms Ω2 and Ω encourages prediction-level and label-level local consistency respectively. and are defined as follows
(P((Q) --> Sij Pa
(4)
(,j &
(4) - Σ Sij L Qa Qjk = --
(5)
where @ indicates element-wise multiplication. To be more specific, ΩP encourages assigning larger weight to Qii and Qyı if the i-th sample is very close to the j-th sample, and their predictions Psk and P 2 from the k-th class centroid are simultaneously high. Analogously, f2 encourages assign- ing larger weight to those samples whose neighborhood label consistency is rather high. Unlike the formulation proposed in [ML [[6], which focuses on sample-to-sample mapping, our method intro- duces sample-to co-aass mapping that leverages the intrinsic coherence structure within the samples.
4.2 Curriculum and Structure-Awant Optimal Transport for Denoising and Relabeling
In the early stages of training or in scenarios with a high noise ratio, the predictions and feature representation would be vague and thus lead to the wrong assignments for SOT. For the purpose of robust clean label identification and corrupted label correction, we further propose a Curriculum and Structure-awee aware Optimal Transport (CSOT), which constructs a robust curriculum allocator. This curriculum allocator gradually selects fraction of the samples with high confidence from the noisy training set, controlled by budget factor, then assigns reliable pseudo labels for them.
Our proposed CSOT for denoising and relabeling is formulated by introducing new curriculum con- straints based on SOT in Problem (E)). Given curriculum budget factor m E [0, 1], our CSOT seeks optimal coupling matrix Q by minimizing following objective
min (- log P, Q) + K (Ω. (Q)+ + (Q)())) Q
(6)
1 Qlc S D 1B. Q1 1g " T
s.t. Q
Q
BxC R
Unlike SOT, which enforces equality constraint on the samples, CSOT relaxes this constraint and defines the total coupling budget as m [0, 1], where m represents the expected total sum of Q. Intuitively speaking, m = 0.5 indicates that top 50% confident samples are selected from all the classes, avoiding only selecting the same class for all the samples within mini-batch. And the budget progressively increases during the training process, shown in Fig. M.
Based on the optimal coupling matrix Q solved from Problem (G), we can obtain pseudo label by argmax operation, i.e. Di = argmax, Qij- In addition, we define the general confident scores of samples as W = [Wo, ".. wB-1). where w;
(m//mm// Since our curriculum allocator assigns weight only fraction samples controlled by m. we use topK(S,k) operation (return top-k indices of input set S) to identify selected samples denoted as 5,
[1, E topk(M, [mB]) (Pp])W)LBB))
6, = (0, otherwise
where [.] indicates the round down operator. Then the noisy dataset Dirain can be splited into Delean and D Peorrupted as follows
Delean + {xx,, w:) 0: = yi, d; 1x;, 21) E Dtrain]
(8)
D Porrupted +- {(x,, Gi. wt 1i # ye, (xi, vi) E Derain
4.3 Training Objectives
To avoid error accumulation in the early stage of training, we adopt two-stage training scheme. In the first stage, the model is supervised by progressively selected clean labels and self-supervised by unselected samples. In the second stage, the model semi-supervised by all denoised labels. No- tably, construct our training objective mainly based on Mixup loss L and Label consistency
5

---

loss Clab same as NCE |55, and a sulf--suer loss Caimsiam proposed in SimSiam [T]]] The detailed formulations of mentioned loss and training process are given Appendix. Our two-stage training objective can be constructed as follows
L
-Detean
+
L
Detean
+ X Caimsiam Dearrupted
C Detean
+
Clab
Delcan
+ X2 Clob
Clab
Deorraphod"
(10)
5 Lightspeed Computation for CSOT
The proposed CSOT new OT formulation with nonconvex objective function and curriculum constraints, which cannot be solved directly by classical OT solvers. To this end, develop lightspeed computational method that involves a scaling iteration within a generalized conditional gradient framework solve CSOT efficiently. Specifically, ation for solving the OT problem with curriculum constraints without considering the local coherent regularized terms, Le.. Curriculum OT (COT). Then, extend our approach to solve the proposed CSOT problem, which involves a nonconvex objective function and curriculum constraints.
first introduce efficient scaling iter-
5.1 Solving Curriculum Optimal Transport
For convenience, we formulate curriculum constraints in Probelm (6) in more general form. Given two vectors a and /3 that satisfy ||a]| 2 || = m, a general polytope of curriculum constraints (a, (3 is formulated as
(a, Q lRle Q1 a Qa,Q1 =Ø]
(11)
For the efficient computation purpose, we consider an entropic regularized version of COT
min (C, + (Q, log Q)
(12)
(a.F
where denote the cost matrix C log P in Probelm 6 for simplicity. In-
spired by [8]. Problem (l2) can be easily re-written as the Kullback-Leibler (KL) projection: mingerI <Kl c/c). Besides, the polytope (a, can be expressed as an intersec- tion of two convex but not affine sets, i.e.
C = Q R
laxx I8] Q11a S a and C2 = Q R
Q Q = 3).
(13)
In light of this, Problem (12) can be solved by performing iterative KL projection between C1 and C2, namely Dykstra's s algorithm [2II] shown in Appendix.
Lemma 1. (Efficient scaling iteration for Curriculum OT) When solving Problem (72) by iterating
Dykstra's 's algorithm, the matrix Q (1) at n iteration is diagonal scaling of K = Cje which the element-wise exponential matrix of -C/e:
(n) = diag (u( Kdiag (v(1))
(14)
where the vectors (m) glal, ((n) E glÅl satisfy v00) ||s and follow the recursion formula
3
(n) min Ko(n-i)
1 and (n)
(15)
K u (n)
The proof given in the Appendix. Lemma || allows fast implementation of Dykstra' 's algorithm by only performing matrix- vector multiplications. This scaling iteration for entropic regularized COT is very similar to the widely-used and efficient Sinkhorn Algorithm [I9]] as shown in Algorithm III
5.2 Solving Curriculum and Structure-Awant Optimal Transport
In the following, we propose to solve CSOT within a Generalized Conditional Gradient (GCG) algorithm [59] framework, which strongly relies on computing Curriculum OT by scaling iterations

---

Algorithm Efficient scaling iteration for entropic regularized Curriculum OT
1: Input: Cost matrix C, marginal constraints vectors a and B, entropic regularization weight
2: Initialize: K t- e C/cs ((0)
|
3: Compute: K + di
K
// Saving computation
diag'
4: for 2,2,3, do
5: (n))) f- min
1oo K. Ki a1)
(1)) 11a) Kzu(n
6:
7: end for
8: Return: diag(u' (diag(v(())
in Algorithm The conditional gradient algorithm [27] 50 has been used for some penalized OT
problems [24], I or nonconvex Gromov- Measev-Wasserstein distances [58, 67 [3] which can be used to
solve Problem (E) directly.
For simplicity, we denote the local coherent regularized terms as I2(-) := (.)(.) + (-). and give an entropic regularized CSOT formulation as follows:
min (a.)(
(C, +Q((Q) + + (Q, log Q)
(16)
Since the local coherent regularized term (P(() differentiable, Problem (IGG) be solved within the GCG algorithm framework, shown in Algorithm D And the linearization procedure in Line be computed efficiently by the scaling iteration proposed in Sec 5.
Algorithm 2 Generalized conditional gradient algorithm for entropic regularized CSOT
1: Input: Cost matrix C, marginal constraints vectors a and B, entropic regularization weight e, local coherent regularization weight k, local coherent regularization function : R|a[x|[] + R, and its gradient function VΩ [gla ||
2: Initialize: Q < a3"
3: for i = 1, 3. do
4: G( + Q + (VI(Q() // Gradient computation
5: Q + argninq:TT(..s) (Q, ( - (Q, Q)
// Linearization, solved efficiently by Algorithm m
6: Choose (3))) E [0, 1] so that it satisfics the Armijo rule // Backtracking line-search
7: Q(++1) - (1- ()) Q(1 + ((()) // Update
8: end for
9: Return: Q(+)
6 Experiments
6.1 Implementation Details
We conduct experiments three standard LNL benchmark datasets: CIFAR-10 BSL CIFAR-100 P3 and Webvision [_.]] We follow most implementation details from the previous work DivideMix
20 and NCE [A5]. Here provide some specific details of our approach. The warm-up epochs
are set to 10/30/10 for CIFAR- 10/110//100// 00/Wevvisio respectively. For CIFAR-10/100 the supervised learning epoch Teup is
250, and the semi-supervised learning epoch Tsemi is set to 200. For Webvision, Teup = 80 and T Tsemi = 70. For all experiments, we set \ = 1, X2 = 1, = 0.1, = And we adopt a simple linear ramp for curriculum budget, i.e. m = min(1.0, 0, mo + - with an initial budget ma = 0.3. For the GCG algorithm, the number of outer loops is set to 10, and the number for inner scaling iteration set to 100. The batch size B for denoising and relabeling is set to 1024. More details will be provided in Appendix.

---

Table 1: Comparison with state-of-the-art methods in accuracy (%) on CIFAR-10 and CIFAR-100. The results are mainly copied from [.S, 98]. We present the performance of our CSOT method using the "mean variance" format, which is obtained from trials with different seeds.
Daraset
CIFAR-10
CIFAR-100
Noise type
Symmetric
Assymetric
Symmettic
MethodNoise rahio
0.2
0.5
0.8
0.9
0.4
0.2
0.5
0.8
0.9
Conss-Ftrroyy
42.7
19.9
F-correction [+
86.8
79.8
63.3
42.9
87.2
61.5
46.6
19.9
10.2
Co-teaching+
89.5
85.7
67.4
47.9
65.6
51.8
27.9
13.7
PENCIL [+:
92.4
89.1
77.5
58.9
88.5
69.4
57.5
31.1
15.3
DivideMix [::
96.1
94.6
93.2
76.0
93.4
71.3
74.6
60.2
31.5
ELR
78.7
33.4
NGC
95.9
94.5
91.6
80.5
90.6
79.3
75.9
62.7
29.8
RRI [8
96.4
95.3
93.3
77.4
92.6
80.3
76.0
61.1
33.1
92.0
UniCon [1]
96.0
95.6
93.9
90.8
94.1
78.9
77.6
63.9
44.8
NCE
96.2
95.3
93.9
88.4
94.5
81.4
76.3
64.7
41.1
Cleaner [X]
91.4
85.4
$6.9
67.4
58.9
31,2
OT-Filter
96.0
95.3
94.0
90.5
95.1
76.7
73.8
61.8
42.8
(Best)
96.6=0.10
96.2+-011
94.4:6.16
90.7=0.33
95.5+:0.06
77.9=00188
67.8:+0.33
50.5=+0446
CSOT (Last)
96.420.18
96.02:0.11
94.3220.0
90.5:0036
95.2::.12
80.2::0.1
17.7=0.14
67.6:20.36
50.3:20.3
Table 2: Comparison with SOTA methods in top-1 5 test accuracy (%) on the Webvision and ImageNet ILSVRCI2 validation sets.
Table 3: Time cost (s) for solving CSOT op- timization problem of different input sizes. VDA indicates vanilla Dykstras algorithm- based CSOT solver, while ESI indicates the effi- cient scaling iteration- -baeedd solver.
Webvision
ILSVRC12
Method
top-1
top-5
top-1
top-5
F-correction [X]]
61.12
82.68
57.36
82.36
Decoupling [
62.54
84.74
58.26
82.26
MentorNet [1]]
63.00
81.40
57.80
79.92
([or), (
VDA based
ESI-based (Ours)
Co-teaching [3]]]
63.58
85.20
61.48
84,70
(1024,10)
0.83
0.82
DivideMix
[+
77.32
91.64
75,20
90.84
(1024,50)
1.00
0.80
ELR [XI]
76.26
91.26
68.71
87.84
(1024,100)
0.87
0.80
EIR++ [S]]
77.78
91.68
70.29
89.76
(50,50)
0.82
0.79
NGC 72
79.20
91.80
74,40
91.00
(100,100)
0.88
0.80
RRL HS
77.80
91.30
74.40
90.90
(500,500)
0.88
0.87
UniCon
[1
77.60
93.44
75.29
93.72
MOIT 55
77.90
91.90
73.80
91.70
(1000,1000)
0.94
0.81
NCE [S
79.50
93.80
76.30
94.10
(2000,2000)
2.11
0.98
(3000_3000)
3.74
0.99
CSOT
79.67
91.95
76.64
91.67
6.2 Comparison with the State-of-the-Arts
Synthetic noisy datasets. Our method validated two synthetic noisy datasets, i.e. CIFAR- 10 [33] and CIFAR-100 [43]. Following [AG, A5], we conduct experiments with two types of label noise: symmetric and asymmetric. Symmetric noise injected by randomly selecting percentage of samples and replacing their labels with random labels. Asymmetric noise is designed to mimic the pattern real-world label errors. i.e. labels are only changed to similar classes (e.g. cat++dog). As shown in Tab, MI our CSOT has surpassed all the state-of-the works across most of the noise ratios. In particular, our CSOT outperforms the previous sfate-of-the-an method NCE [45] by 2.3%. 3.1% and 9.4% under high noise rate of CIFAR- sym-0.8, CIFAR- 100 sym-0.8/0.9, respectively.
Real-world noisy datasets. Additionally, we conduct experiments on a large-scale dataset with real-world noisy labels, i.e. Web Vision [29]] WebVision contains 2.4 million images crawled from the web using the 1,000 concepts in ImageNet ILSVRC12 [20]]. Following previous works [36, [5]] we conduct experiments only using the first 50 classes of the Google image subset for a total of ~61,000 images. As shown in Tab. A our CSOT surpasses other methods in top-1 accuracy on both Webvision and ILSVRC12 validation sets, demonstrating its superior performance in dealing with real-world noisy datasets. Even though NCE achieves better top-5 accuracy, suffers from high time costs (using single NVIDIA A100 GPU) due to the co-training scheme, shown in Tab. SS

---

Table 4: Ablation studies under multiple label noise ratios on CIFAR-10 and CIFAR-100. "repl." is an abbreviation for "replaced", and _ represents a cross-entropy loss. GMM refers to the selection of clean labels based on small-loss criterion [_6]]. CT (confidence thresholding [62]] is a relabeling scheme where we set the CT value to 0.95.
Dataset
CIFAR-I0 10
CIFAR- 100
Noise type
Sym.
Asym.
Sym.
Avg
Method/Noise
0.5
0.9
0.4
0.5
0.9
(a) Classical OT
95.45
91.95
82.35
95.04
75.96
43.28
78.07
Denoise
(b) Structure-awe aware OT
95.86
91.87
83.29
95.06
76.20
63.73
44.57
78.65
Relaheling
(c) CSOT w/ QP and Q
95.53
93.84
89.50
95.14
75.96
66,50
47.55
80.57
Technique
(d) CSOT w/o f?"
95.77
94.08
89.97
95.35
76.09
66.79
48.13
80.88
(c) CSOT w/o Q
95.55
93.97
90.41
95.15
76.17
67.28
48.01
80.93
(f) GMM + 2
92.48
80,37
31.76
90.80
69.52
48.49
20.86
62.04
Learning Technique
(g) CSOT repl. C with g
93.47
81.93
53.45
91.43
72.66
50.62
21.77
66.48
CSOT w/o C
95.34
93.04
88.9
94.11
75.16
61.13
36.94
77.80
(i) CSOT repl. correction with CT (0.95)
95.46
90.73
89.09
95.21
75.85
64,28
48.76
79.91
( CSOT w/o D
Pemerspted
95.92
94.17
89.31
95.16
76.38
66.17
45.56
80.38
CSOT
96.20
94.39
90.65
95.50
77.94
67.78
50.50
81.85
6.3 Ablation Studies and Analysis
Effectiveness of CaOT-based denoising and relabeling. To verify the effectiveness of each com- ponent in our CSOT, we conduct comprehensive ablation experiments, shown in Tab. EL Compared to classical OT, Structure-aware OT, and Curriculum OT, our proposed CSOT has achieved supe- rior performance. Specifically, our proposed local coherent regularized terms Ω. and f2 indeed contribute to CSOT, as demonstrated in Tab (e). Furthermore, our proposed curriculum con- straints yield an improvement of approximately 2% for both classical OT and structure- OT, as shown in Tab g (a)( Particularly, under high noise ratios, the improvement can reach up to 4%, which demonstrates the effectiveness of the curriculum relabeling scheme.
Effectiveness of clean labels identification via CSOT. As shown in Tab. E (f), replacing our CSOT-based denoising and relabeling with GMM [+6]] for clean label identification significantly degrades the model performance. This phenomenon can be explained by the clean accuracy during training (Fig. ( and clean recall rate (Fig. 20), in which our CSOT consistently outperforms other methods in accurately retrieving clean labels, leading to significant performance improvements. These experiments fully show that CSOT can maintain both high quantity and high quality of clean labels during training.
Effectiveness of corrupted labels correction via CSOT. As shown in Tab. E (h), only training with identified clean labels leads to inferior model performance. Furthermore, replacing our CSOT- based denoising and relabeling with confidence thresholding (CT) [62]] for corrupted label correction also degrades the model performance, as shown in Tab. g (i). The CT methods assign pseudo labels to samples based on model prediction, which is unreliable in the early training stage, especially under high noise rates. Our CSOT-based denoising and relabeling fully consider the inter- and intra- distribution structure of samples, yielding more robust labels. Particularly, our CSOT outperforms NCE and DivideMix significantly in label correction, as demonstrated by the superior corrected accuracy in Fig. 26 and the improved clarity of the confusion matrix Fig. S7
Effectiveness of curriculum training scheme. According to the progressive clean and corrupted accuracy during the training process shown in Fig. and Fig. ME, our curriculum identification scheme ensures high accuracy in the early training stage, avoiding overfitting to wrong corrected labels. Note that since our model trained using only a fraction of clean samples, it is crucial to employ powerful supervised learning loss to facilitate better learning. Otherwise, the performance will be poor without the utilization of powerful supervised training loss, as evidenced in Tab. g (g). In addition, the incorporation of self-supervised loss enhances noisc-robust representation, particularly in high noise rate scenarios, as demonstrated our experiments in Tab. E (i).
Time cost discussion for solving CSOT To verify the efficiency of our proposed lightspeed seal- ing iteration, we conduct some experiments for solving CSOT optimization problem of different input sizes on single GPU NVIDIA A100, As demonstrated in Tab. B. our proposed lightspeed

---

Acorscy
Acceas cometed
06-
02
04
Texting meggrno
(a) Clean accuracy
(b) Corrected accuracy
(c) Clean recall rate
Figure 2: Performance comparison for clean label identification and corrupted label correc- tion.
computational method that involves an efficient scaling iteration (Algorithm M achieves lower time cost compared vanilla Dykstras algorithm (Algorithm (SG). Specifically, compared the vanilla Dykstra-base based approach, our efficient scaling iteration version can achieve a speedup of to 3.7 times, thanks to efficient matrix-vector multiplication instead matrix-matrix multiplication. More- over, even for very large input sizes, the computational time cost does not increase significantly.
7 Conclusion and Limitation
In this paper, we proposed Curriculum and Structure-awar Optimal Transport (CSOT), novel so- lution to construct robust denoising and relabeling allocator that simultaneously considers the inter- and intra-distribution structure of samples. Unlike current approaches, which rely solely on the model's predictions, CSOT considers the global and local structure of the sample distribution construct a robust denoising and relaheling allocator. During the training process, the allocator as- signs reliable labels to fraction of the samples with high confidence, ensuring both global discrim- inability and local coherence. To efficiently solve CSOT, we developed a lightspeed computational method that involves scaling iteration within a generalized conditional gradient framework. Exten- sive experiments on three benchmark datasets validate the efficacy of our proposed method. While dlas-imbalanne cases are not considered in this paper within the context of LNL, believe that our approach can be further extended for this purpose.
8 Acknowledgement
This work was supported by NSFC (N..33303,,9, Shanghai Sailing Program (21YF1429400, 22YF1428800), Shanghai Local College Capacity Building Program (23010503100) Shanghai Frontiers Science Center of Human-centeree Artificial Intelligence (ShangHAI), MoE Key Labo- ratory of Intelligent Perception and Human Machine Collaboration (ShanghaiTech University), and Shanghai Engineering Research Center of Intelligent Vision and Imaging.
10

---

References
[1] David Alvarez-Melis, Tommi Jaakkola, and Stefanie Jegelka. Structured optimal transport. In Interna- tional conference on artificial intelligence and statistics, pages 1771-1780. PMLR, 2018.
[2] Shuang Ao. Tianyi Zhou, Guodong Long, Qinghua Lu, Liming Zhu, and Jing Jiang. Co-pilot: Collab- orative planning and reinforcement learning on sub-task curriculum. Advances in Neural Information Processing Systems,
10444-10456, 2021.
[3] Eric Arazo, Diego Ortego, Paul Albert, Noel OConnor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In International conference on machine learning, pages 312-321. PMLR, 2019.
[4] Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at mem- orization in deep networks. In International conference on muchine learning, pages 233-242. PMLR, 2017.
[5] YM Asano, Rupprecht, and A Vedaldi. Self-labelling via simultaneous clustering and representation learning. In International Conference on Learning Representations, 2019.
[6] Dara Bahri, Heinrich Jiang, and Maya Gupta. Deep k-nn for noisy labels. In International Conference on Machine Learning, pages 540-550. PMLR, 2020.
[7] Heinz H Bauschke and Adrian S Lewis. Dykstras algorithm with bregman projections: A convergence proof. Optimization, 48(4):409-477) 2000-422220000 2000.
[8] Jean-David Benamou, Guillaume Carlier, Marco Cuturi, Luca Nenna, and Gabriel Peyré. Iterative bregman projections for regularized transportation problems. SIAM Journal on Scientific Computing, 37())A1111-A1133 2015.
[9] Dimitri P Bertsekas. Nonlinear programming. Journal of the Operational Reseurch Society, 48(3)3334
334, 1997.
[10] Lev M Bregman. The relaxation method of finding the common point of convex sets and its application to the solution of problems convex programming. USSR computational mathematics and mathematical physics, 7(3):200-21) 217, 1967.
[11] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. Advances neural information processing systems, 33:9912-9924, 2020.
[12] Wanxing Chang, Ye Shi, Hoang Tuan, and Jingya Wang. Unified optimal transport framework for univer- sal domain adaptation. Advances in Neural Information Processing Systems, 35:29512-29524, 2022.
[13] Laetitia Chapel. Mokhtar Z Alaya, and Gilles Gasso. Partial optimal tranport with applications on positive- unlabeled learning. Advances in Neural Information Processing Systems, 33:2903-291 2020.
[14] Pengfei Chen, Ben Ben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In International Conference on Machine Learning, pages 1062-1070 PMLR, 2019.
[15] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IFFF/CVF conference on computer vision and pattern recognition, pages 15750-15758, 2021.
[16] Ching- Yao Chuang, Stefanie Jegelka, and David Alvarcz-Melis. Infoot: Information maximizing optimal transport. In International Conference Machine Learning, pages 222222242. PMLR, 2023.
[17] Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamonjy. Optimal transport for domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39:1853-1888 1865, 2017.
[18] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan. and Quoe V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1 13--133, 2019.
[19] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26:2292 2300, 2013.
[20] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchi- cal image database. In 2009 IEF.E conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[21] Richard L Dykstra. An algorithm for restricted least squares regression. Journal of the American Statisti- cal Association, 78(38))837-842, 1983.
[22] Erik Englesson and Hossein Azizpour. Consistency regularization can improve robustness to label noise. In International Conference on Machine Learning Workshops, 2021 Workshop on Uncertainty and Ro- bustness in Deep Learning, 2021.

---

[23] Chuanwen Feng. Yilong Ren, and Xike Xie. Ot-filter: An optimal transport filter for learning with noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16164-16174, 2023.
[24] Sira Ferradans, Nicolas Papadakis, Gabriel Peyré, and Jean- François Aujol. Regularized discrete optimal transport. SIAM Journal on Imaging Sciences, 7(3):1833- 188-1882, 2014.
[25] Enrico Fini, Enver Sangineto. Stéphane Lathuilière, Zhun Zhong, Moin Nabi, and Elisa Ricci. A uni- lied objective for novel class discovery. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9284-9292, 2021.
[26] Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélic Boishunon, Stanislas Chambon, Lactitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalic T.H. Gayraud, Hicham Janati, Alain Rakotomamoniy, levgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong. and Titouan Vayer. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78))1-8, 2021.
[27] Marguerite Frank and Philip Wolfe. An algorithm for quadratic programming. Naval research logistics quarterly, 3(1-2)99-1110, 1956.
[28] Dandan Guo, Zhuo Li. He Zhao, Mingyuan Zhou, Hongyuan Zha, et al. Learning to re-weight examples with optimal transport for imbalanced classification. Advances in Neural Information Processing Systems, 35:25517-2555 255300, 2022.
[29] Lan-Zhe Guo and Yu-Feng Li. Class-inbalanced supe-supe learning with adaptive thresholding. In International Conference on Machine Learning, pages 8082-8094, PMIR, 2022.
[30] Sheng Guo, Weilin Huang, Haozhi Zhang, Chenfan Zhuang. Dengke Dong, Matthew R Scott, and Ding- long Huang. Curriculumnet: Weakly supervised learning from large-scale web images. In Proceedings of the European conference on computer vision, pages 135- 2018,
[31] Bo Han. Quamming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31, 2018.
[32] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEFE international conference on computer vision, pages 2961-2969, 2017.
[33] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on compuder vision and pattern recognition, pages 770-778, 2016.
[34] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Eurupean Conference on Computer Vision, pages 630-645. Springer, 2016.
[35] Dan Hendrycks, Mantas Mazeika, Dunean Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. Advances in neural information processing systems, 31, 2018.
[36] Martin Jaggi. Revisiting wolfe: Projection free sparse convex optimization. In International con- ference on machine learning, pages 427-435. PMIR, 2013.
[37] Lu Jiang, Di Huang, Mason Liu, and Weilong Yang. Beyond synthetic noise: Deep learning on controlled noisy labels. In International conference on machine learning, pages 4804-4815 15. PMLR, 2020.
[38] Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander Hauptmann. Self-paced curriculum learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
[39] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International conference on machine learning, pages 2304-2313. PMLR, 2018.
[40] Leonid V Kantorovich. On the translocation of masses. Journal of mathematical sciences, 133(4):1381- 1382, 2006.
[41] Nazmul Karim, Mamshad Nayeem Rizve, Nazanin Rahnavard, Ajmal Mian, and Mubarak Shah. Uni- con: Combating label noise through uniform selection and contrastive learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 9676-9686. 2022.
[42] Faisal Khan, Bilge Mutlu, and Jerry Zhu. How do humans teach: On curriculum learning and teaching dimension. Advances in neural information processing systems, 24, 2011.
[43] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
[44] Zhengfeng Lai, Chao Wang. Sen-ching Cheung, and Chen-Nee Chuah. Sar: Self-adaptive refinement pseudo labels for
semi-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4091-4 1990- 2022.
12

---

[45] Jichang Li, Guanbin Li, Feng Liu. and Yizhou Yu. Neighborhood collective estimation for noisy label identification and correction. In European Conference Computer Vision, pages 128-145. Springer. 2022.
[46] Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. In International Conference on Learning Representations, 2019.
[47] Junnan Li. Yongkang Wong, Qi Zhao, and Mohan S Kankanhalli. Learning to learn from noisy labeled data In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 5051-5059, 2019.
[48] Junnan Li, Caiming Xiong, and Steven CH Hoi. Learning from noisy data with robust representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9485- 9494, 2021.
[49] Wen Li. Limin Wang. Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data. CoRR. 2017.
[50] Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez- Grandaa Early-learning reg- ularization prevents memorization of noisy labels. Advances in neural information processing systems, 33:20331-20342, 2020.
[51] Zhanyu Ma and Ame Leijon. Bayesian estimation of beta mixture models with variational inference. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(11):2160-25 2173, 2011.
[52] Eran Malach and Shai Shalev- Shwaev-Swwrrtz. Decoupling" when update" from" how to update". Advances in neural information processing systems, 30, 2017.
[53] Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Cur- riculum learning for reinforcement learning domains: A framework and survey. The Journal of Machine Learning Research, 21(1):7382 14312, 2020.
[54] Vu Nguyen, Sachin Farfade, and Anton van den Hengel. Confident sinkhorn allocation for pseudo- labeling. arXiv preprint XrXiv:2206. 05880, 2022.
[55] Diego Ortego, Eric Arazo, Paul Albert, Noel E O' O'Connor, and Kevin McGuinness. Multi-objective inter- polation training for robustness to label noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Partern Recognition, pages 6606-6615, 2021.
[56] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1944 1952, 2017.
[57] Haim Permuter, Joseph Francos, and Ian Jermyn. A study of gaussian mixture models of color and texture features for image classification and segmentation. Pattern recognition, 39(4):695-708 906, 2006.
[58] Gabriel Peyré, Marco Cuturi, and Justin Solomon. Gromov-wasserstinn averaging of kernel and distance matrices. In International conference on machine learning, pages 2664-2677 PMLR, 2016.
[59] Alain Rakotomamonjy, Rémi Flamary, and Nicolas Courty. Generalized conditional gradient: analysis of convergence and applications. arXiv preprint 170.0:1570 2015.
[60] Mengye Ren. Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep lcarning. In International conference machine learning, pages 4334-4343. PMLR, 2018.
[61] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
[62] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Do- gus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-608 2020.
[63] Hwanjun Song. Minseok Kim, and Jae-Gil Lee. Selfie: Refurbishing unclean samples for robust deep learning. In International Conference on Machine Learning, pages 5907-5915. PMLR, 2019.
[64] Christian Szegedy. Sergey loffe, Vincent Vanhoucke, and Alexander Alemi. Inception-v4, inception- resnet and the impact of residual connections on learning. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
[65] Kai Sheng Tai. Peter D Bailis, and Gregory Valiant. Sinkhorn label allocation: Semi-supervvsed classifi- cation via annealed self-training. In International Conference on Machine Learning, pages 10065-10075 PMLR, 2021.
[66] Laurens Van der Maaten and Geoffrey Hinton, Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[67] Titouan Vayer. Lactitia Chapel, Rémi Flamary, Romain Tavenard, and Nicolas Courty. Fused gromov- wasserstein distance for structured objects. Algorithmss, 13(9)2112, 2020.
13

---

[68] Haobo Wang, Mingxuan Xia, Yixuan Li, Yuren Mao, Lei Feng, Gang Chen, and Junbo Zhao. Solar: Sinkhorn label refinery for imbalanced partial-label learning. Advances in Neural Information Processing Systems, 13:8104-817, 2022.
[69] Yidong Wang. Hao Chen, Qiang Heng, Wenxin Hou, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj. Zhen Wu, and Jindong Wang. Freematch: Self-adaptive thresholding for semi-supervised learning. International Conference on Learning Representations. 2022.
[70] Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey, Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 322-330, 2019.
[71] Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 13726-13735, 2020.
[72] Zhi-Fan Wu, Tong Wei, Jianwen Jiang, Chaojie Mao, Mingqian Tang, and Yu-Feng Li. Nge: A uni- fied framework for learning with open-world noisy data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 62-71, 2021.
[73] Jun Xia, Cheng Tan, Lirong Wu, Yongjie Xu, and Stan Li. Ot cleaner: Label correction as optimal transport. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 3953- 3957. IEEE, 2022.
[74] Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2691-2699, 2015.
[75] Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi- supervised learning with dynamic thresholding. In International Conference on Machine Learning, pages
11525-11536. PMLR, 2021.
[76] Kun and Jianxin Wu. Probabilistic end-to-end noise correction for learning with noisy labels. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 7017- 7025, 2019.
[77] Xingrui Yu, Bo Han, Jiangehao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagree- ment help generalization against label corruption? In International Conference on Machine Learning, pages 7164-711 7173. PMLR, 2019.
[78] Bowen Zhang. Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang. Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems, 34:1 144118088-8414 2021.
[79] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep leamnngg (still) requires rethinking generalization. Communications of the ACM, 64()):07--155 2021.
[80] Yaig Zhang, XiMing Xing, and Liang Liu. Dualgraph: A graph-based method for reasoning about label noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9654-9663, 2021.
[81] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. International Conference Learning Representations, 2017.
[82] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018.
[83] Kecheng Zheng, Wu Liu, Lingxiao He, Tao Mei, Jiebo Luo, and Zheng-Jun Zha. Group-aware label transfer for domain adaptive person re-identitication. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 5310-5319, 2021.
[84] Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Curriculum learning by optimizing learning dynamics. In International Conference on Artificial Intellignnce and Statistics, pages 433-441, PMLR, 2021.
[85] Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Robust curriculum learning: from clean label detection noisy label self-correctton. In International Conference on Learning Representations, 2021.
14


Reviews:
[{"review_id": "N2KyGsxLY6", "reviewer": "Reviewer_qnim", "summary": "This paper proposes a new noisy label learning approach based on Optimal Transport (OT) and Pseudo-Labeling (PL). Specifically, the authors extent OT-based PL with the consideration of the intrinsic coherence structure of sample distribution. Consequently, this paper proposes a novel optimal transport formulation, namely Curriculum and Structure-aware Optimal Transport (CSOT), which constructs a robust denoising and relabeling allocator that mitigates error accumulation. Experiments on both controlled and real noisy label datasets show the effectiveness of the proposed method.", "questions": "Please refer to Weaknesses", "limitations": "N/A.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a method named Curriculum and Structure-aware Optimal Transport (CSOT) to address the problem of noisy label learning, and the application of OT-based pseudo-labeling in tackling noisy label learning problem has not been thoroughly investigated.\n\n2. The experimental results on different datasets in this paper validate the effectiveness of the proposed method. Additionally, several ablation experiments are conducted to demonstrate the effectiveness of each module in the method.", "weaknesses": "1. In terms of methodological novelty, OT-based PL has been previously applied to other problems, and this paper only applies it to the specific problem namely noisy label learning rather than introducing it for the first time. Additionally, employing curriculum learning to address the issue of pseudo-labeling is a common strategy in the field of weakly supervised learning.\n\n2. The utilization of SOT is a key contribution of this paper. However, the current motivation behind this aspect, as presented in Figure 1, is not sufficiently clear. The authors are encouraged to provide additional descriptions in this section to enhance the clarity and understanding of the motivation.\n\n3. Since the differences between the proposed method and the comparison methods in several cases is too small, it is hard to provide a clear comparison without the present of standard deviations. Additionally, the backbone and other parameter settings of the SOTA methods are not clearly listed, thus, further evidence is needed to establish the fairness of the comparisons.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments which helped us improve the quality of our work. In the following, we have provided a point-by-point response to the comments.\n\n\n> **Q1**. OT-based Pseudo-Labeling (PL) is not proposed for the first time and curriculum learning to address the issue of pseudo-labeling is a common strategy.\n\n**A1**. Firstly, we emphasize that directly applying the original OT to certain PL tasks may result in sub-optimal performance. Consequently, it **becomes necessary for researchers to investigate more adaptive OT formulations for specific problems** [9][54], including off-the-shelf variants like unbalanced OT and partial OT, to address the specific challenges of the problem. To this end, we propose a novel CSOT formulation tailored for the denoising and relabeling task. Notably, our CSOT formulation innovatively incorporates two local coherent regularization terms and curriculum constraints, enabling the incremental generation of reliable pseudo-labels for the Learning with Noisy Labels task.\n\nSecondly, we would like to claim that **our work is the first to propose a curriculum scheme that fully considers the inter- and intra-distribution structure of the samples based on OT**, to the best of our knowledge. Also, introducing curriculum scheme to OT requires a new solver and **we innovatively propose a lightspeed computational method**. This approach stands in stark contrast to existing curriculum-based PL methods, setting our work apart in terms of its novelty and uniqueness.\n\nThirdly, we highlight that our work **provides a valuable example of developing a customized OT formulation and a corresponding solver**. By showcasing our method's adaptability and efficacy, we aim to contribute to the wider application and exploration of OT across various domains.\n\n\n> **Q2**. Provide additional descriptions about the Structure-aware OT in Figure.\n\n**A2**. Thanks for your constructive suggestion. We will add the following descriptions to Figure 1 in our final version.\n\"(Top) Comparison between classical OT and our proposed Structure-aware OT. Classical OT tends to mismatch two nearby samples to two far-away class centroids when the decision boundary is not accurate enough. To mitigate this, our SOT generates local consensus assignments for each sample by preserving prediction-level and label-level consistency. Notably, for vague samples located near the ambiguous decision boundary, SOT rectifies their assignments based on the neighborhood majority consistency.\"\n\nMoreover, to further show the differences among prediction-, OT-, and Structure-aware OT-Based PL, we provided a more intuitive illustration in Figure S3 (Appendix B.1).\n\n\n\n\n\n> **Q3**. Lack of standard deviations and experimental settings of the SOTA methods.\n\n**A3**. Thank you for raising the concern. For the standard deviations, we did provide the results of standard deviations on CIFAR10/100 in Table 1. Here we provide standard deviations for the Webvision dataset as follows: \n\nTable 1. Comparison between NCE and CSOT in top-1/5 test accuracy (\\%) on the\nWebvision and ImageNet ILSVRC12 validation sets.\n|        |Webvision-top-1|Webvision-top-5|ILSVRC12-top-1|ILSVRC12-top-5|\n|  ----  | ----  | ----  | ----  | ----  |\n|CSOT (Ours)| 79.67±0.14 | 91.95±0.21 | 76.64±0.16 | 91.67±0.18 |\n\nFor the experimental settings, we inherit the same backbone, optimizer parameters from the SOTA works NCE and DivideMix. More details are provided in Appendix A.1, please refer to our supplementary materials."}, {"review_id": "ABosKS430T", "reviewer": "Reviewer_gpRK", "summary": "This paper proposes a novel optimal transport formulation, called Curriculum and Structure-aware Optimal Transport (CSOT) for learning with noisy labels. CSOT considers both the inter- and intra-distribution structure of the samples to construct a robust denoising and relabeling allocator. It’s worthing noting that Notably, CSOT is a new OT formulation with a nonconvex objective function and curriculum constraints. The authors developed a lightspeed computational method that involves a scaling iteration within a generalized conditional gradient framework to solve CSOT efficiently. ", "questions": "1. How does CSOT incorporate the global and local structure of the sample distribution? Could you provide more details on the methodology used to capture this information?\n\n2. Can you explain in more detail the nonconvex objective function of CSOT? How does the nonconvexity affect the optimization process, and how does the proposed method handle this challenge?\n\n3. What is the iteration number required by the generalized conditional gradient? \n\n4. Are there any specific scenarios or types of noisy labels where CSOT may not perform as well? \n", "limitations": "CSOT is described as having a nonconvex objective function. Nonconvex optimization problems can be challenging to solve, and they may have multiple local optima, making it difficult to guarantee finding the global optimum. This could potentially impact the reliability and efficiency of the proposed method. ", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. Novel Approach: The paper proposes a novel optimal transport (OT) formulation called Curriculum and Structure-aware Optimal Transport (CSOT) to address the challenge of learning with noisy labels. I believe this paper introduces a new perspective and potentially brings fresh insights to the field.\n\n2. Consideration of Global and Local Structure: Unlike current approaches that evaluate each sample independently, CSOT concurrently considers the inter- and intra-distribution structure of the samples. This consideration of both global and local structure helps construct a more robust denoising and relabeling allocator, potentially leading to improved performance.\n\n3. Incremental Assignment of Reliable Labels: CSOT incrementally assigns reliable labels to a fraction of the samples with the highest confidence during the training process. This approach ensures that the assigned labels have both global discriminability and local coherence, which could contribute to better generalization and reduced overfitting.\n\n4. This paper provides a very detailed derivation for the lightspeed computational method. ", "weaknesses": "Researchers or practitioners interested in using CSOT may need to invest additional effort in adapting or developing specialized solvers. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for our paper's positive feedback and constructive suggestions. Here are our responses to the reviewer's comments.\n\n> **Q1**. Researchers or practitioners interested in using CSOT may need to invest additional effort in adapting or developing specialized solvers.\n\n**A1**. Thank you for raising the concern.\n\nFirstly, We would like to justify that **our proposed solver is available for arbitrary differentiable regularization term $\\Omega$, marginal constraints vectors $\\alpha$, and $\\beta$** in Eq.(16), as we stated in Algorithm 2 Input row (Line 1).\n\nSecondly, **our code provides a versatile solver** (Algorithm 2) API that can be used with any user-defined $\\Omega$, $\\nabla\\Omega$, $\\alpha$, and $\\beta$. This feature simplifies the process for follow-up researchers to conduct their experiments, especially for those who wish to validate the effectiveness of their own customized regularization terms $\\Omega$. As a result, researchers do not have to spend time adapting or developing specialized solvers, thus enhancing the efficiency of their work.\n\nThirdly, we highlight that **our work provides a valuable example of developing a customized OT formulation and a corresponding solver**. By showcasing our method's adaptability and efficacy, we aim to contribute to the wider application and exploration of OT across various domains.\n\n\n> **Q2**. Provide more details about incorporating the global and local structure of the sample distribution in CSOT.\n\n**A2**. Firstly, the local structure, i.e. intra-distribution coherence among samples, is preserved by two local coherent regularized terms defined in Equations (4) and (5). Technically speaking, to formulate the coherent regularized terms, we construct the correlation among $i$-th sample, $j$-th sample, and $k$-th class centroid by element-wise multiplication. As shown in Figure 1 Top, classical OT tends to mismatch two nearby samples to two far-away class centroids when the decision boundary is not accurate enough. To mitigate this, our SOT generates local consensus assignments for each sample by preserving prediction-level and label-level consistency. Notably, for vague samples located near the ambiguous decision boundary, SOT rectifies their assignments based on the neighborhood majority consistency.\n\nSecondly, the global structure, i.e. inter-distribution discriminability between samples and categories, is preserved by curriculum constraints $\\Pi^c(\\alpha,\\beta)$ defined in Eq.(11). This property is inherited from the OT constraints $\\Pi(\\alpha,\\beta)$ defined in Section 3 (Line 122), which enforces the marginal distributions of the coupling matrix equal to given samples and categories distribution $\\alpha$ and $\\beta$. As shown in Figure S3 (Appendix B.1), OT-based Pseudo-Labeling (PL) tends to generate more discriminative labels than prediction-based PL. To explain this, OT-based PL optimizes the mapping problem by considering the inter-distribution matching of samples and categories, rather than the prediction-based PL assigning labels solely in a per-class manner. \n\nThirdly, our proposed curriculum constraints incorporate the global and local structure of the sample distribution, prioritizing samples with better global discriminability and local coherence properties for label assignment, thereby enabling a robust curriculum allocator.\n\n\n> **Q3**. How does the nonconvexity affect the optimization process, and how does the proposed method handle this challenge?\n\n**A3**. The nonconvexity would bring some problems including local minima, rather slow convergence speed than convex case. However, solving nonconvex objectives with the generalized conditional gradient (GCG) algorithm has been supported by strong convergence analysis [R1][R2]. And we find our CSOT converges fast as shown in Figure S5 (Appendix B.4).\n\nMoreover, some OT-like nonconvex problems, such as Gromov-Wasserstein problem, are also solved by GCG algorithm and provide convergence analysis [10][19]. \n\n[R1] Bredies K, Lorenz D, Maass P. Equivalence of a generalized conditional gradient method and the method of surrogate functionals[M]. Bremen, Germany: Zentrum für Technomathematik, 2005.\n[R2] Beck A. First-order methods in optimization[M]. Society for Industrial and Applied Mathematics, 2017.\n\n\n\n> **Q4**. Specify the iteration number required by the generalized conditional gradient.\n\n**A4**. As we specified in Section 6.1 (Line 238-239), the number of outer loops is set to 10, and the number for inner scaling iteration is set to 100.\n\n\n\n> **Q5**. Specify the potential scenarios or types of noisy labels where CSOT may not perform as well.\n\n**A5**. As we stated in Section 7, extreme class-imbalance cases are not considered in this paper. In highly imbalanced scenario, the generated pseudo-labels would be biased due to uniformed class distribution vector $\\beta$. However, we believe that our approach can be further extended for this purpose in the future work. "}, {"review_id": "e9BLgwtKeF", "reviewer": "Reviewer_ZdDn", "summary": "This paper introduces CSOT, an approach to address the challenge of noisy labels in machine learning models. CSOT incorporates optimal transport formulation to assign reliable labels during training, considering the structure of the sample distribution. The authors also propose an efficient computational method for solving CSOT. Experimental results demonstrate the superior performance of CSOT compared to existing methods for learning with noisy labels.", "questions": "- ", "limitations": "The limitations of this paper are summarized in the \"Question\" and \"Weakness\" sections\n------\n(Raise score from 5 to 6 after rebuttal)", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper is strongly motivated by theoretical analysis, particularly optimal transport analysis.\n- The writing style is clear and easy to follow.\n- CSOT exhibits superior performance when compared to previous algorithms.", "weaknesses": "- The paper lacks a comparison with a baseline algorithm called UNICON [1], which has shown good performance in highly noisy scenarios (e.g., 0.9 noisy ratio). It would be valuable for the authors to include a performance comparison with UNICON.\n- The authors do not analyze the case of instance-wise noisy labels, which is a prevalent type of noisy label model. Including an analysis of this case would be beneficial.\n- The paper does not investigate the sensitivity of hyperparameters, which are required to run the algorithm. It would be valuable for the authors to perform a hyperparameter sensitivity analysis.\n- To enable a comprehensive comparison, the authors should report both the best and last performances of the model, as models trained on noisy labels tend to memorize the noisy labels.\n\n[1] UNICON: Combating Label Noise Through Uniform Selection and Contrastive Learning\n\nMinor)\nThe legend size in Figure 2 is too small to read.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments which helped us improve the quality of our work. In the following, we have provided a point-by-point response to the comments.\n\n> **Q1**. Lack of a comparison with a baseline algorithm called UNICON.\n\n**A1**. Thanks for your constructive suggestion. We will add UNICON as one of the baseline methods in our final version. And here is a simplified comparison table for UNICON and our CSOT in CIFAR-10/100 dataset, which shows that **our CSOT still outperforms UNICON on CIFAR 100 under high noise ratios**.\n\nTable 1. Comparison between UNICON and CSOT in test accuracy (\\%) on CIFAR-10.\n|        |sym-20\\%|sym-50\\%|sym-80\\%|sym-90\\%|asym-40\\%|\n|  ----  | ----  | ----  | ----  | ----  |  ----  |\n|UNICON| 96.0 | 95.6 | 93.9 | **90.8** | 94.1  |\n|CSOT (Ours)    | **96.6** | **96.2** | **94.4** | 90.7 |  **95.5**  |\n\nTable 2. Comparison between UNICON and CSOT in test accuracy (\\%) on CIFAR-100.\n|        |sym-20\\%|sym-50\\%|sym-80\\%|sym-90\\%|\n|  ----  | ----  | ----  | ----  | ----  |\n|UNICON| 78.9 | 77.6 | 63.9 | 44.8 |\n|CSOT (Ours)    | **80.5** | **77.9** | **67.8** | **50.5** |\n\n\n\n> **Q2**. Supplement an analysis of the case of instance-wise noisy labels.\n\n**A2**. Thanks for your constructive suggestion. We validate the effectiveness of our proposed method in instance-wise noise under the noise rate of 20\\%, 40\\%, 60\\% in the following table.\n\nTable 3. Comparison of instance-wise noise among DivideMix, NCE, and CSOT in test accuracy (\\%) on CIFAR-10.\n|  noise rate  |20\\%|40\\%|60\\%|\n|  ----  | ----  | ----  | ----  |\n|DivideMix| 92.26 | 93.86 | 54.34 |\n|NCE| 96.00 | 95.24 | 75.58 |\n|CSOT (Ours)    | **96.21** | **95.82** | **76.65** |\n\nTable 4. Comparison of instance-wise noise among DivideMix, NCE, and CSOT in test accuracy (\\%) on CIFAR-100.\n|   noise rate |20\\%|40\\%|60\\%|\n|  ----  | ----  | ----  | ----  |\n|DivideMix| 77.52 | 73.58 | 38.72 |\n|NCE| **80.40** | 74.92 | 70.34 |\n|CSOT (Ours)    | 79.22 | **76.55** | **72.71** |\n\n\n> **Q3**. Supplement the sensitivity of hyperparameters.\n\n**A3**. We analyzed the sensitivity of hyperparameters $\\varepsilon$ and $\\kappa$ in our CSOT in supplementary materials, please refer to Figure S7 and Figure S8 (Appendix B.6).\n\n\n> **Q4**. Supplement both the best and last performances of the model.\n\n**A4**. We have reported both the best and last performances for CIFAR-10/100 dataset in Table 1. Following the existing method, we only report the last performance on the WebVision and Clothing1M dataset.\n\n\n> **Q5**. The legend size in Figure 2 is too small to read.\n\n**A5**. Thank you for pointing this out. We will adjust it to a more readable size in our final version."}, {"review_id": "J6OBpG8zs4", "reviewer": "Reviewer_Zh5c", "summary": "This paper introduces a novel formulation of Optimal Transport (OT), named Curriculum and Structure-Aware Optimal Transport, for generating pseudo labels by considering both inter- and intra-distribution structures of samples. Moreover, to efficiently estimate the distribution's structure, the authors adopt a curriculum paradigm to progressively train the proposed denoising and relabeling allocator. Additionally, they present a computation method for the proposed CSOT that ensures faster processing speeds, reducing computational overhead. Experimentally, this paper achieved SOTA performance on various benchmarks.", "questions": "1.For more precision, I would like to suggest providing a more detailed description of Equation (6). What does the left constraint in Equation (6) signify? Why should $Q\\mathbb{1}_{C}$ not exceed $\\frac{1}{B}\\mathbb{1}_{B}$? Furthermore, it seems that $\\frac{m}{c}\\mathbb{1}_{C}$ is not a simplex. Could you please explain the rationale behind this relationship and provide additional descriptions or citations to support it? Additionally, the introduction of curriculum into SOT appears to be too direct and lacks smoothness.\n\n2.The identification of selected samples in Equation (7) is confusing. What does \"topK(W,⌊mB⌋)\" mean? Based on my understanding, it implies that a sample pair (x_i, y_i) will be considered clean if it belongs to the top k convincing samples in the current training batch. I would suggest adding a description for the round-down symbol ⌊ and ⌋ to clarify its purpose.", "limitations": "The authors describe its limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.Estimating the intra- and inter-structure coherence of samples is a convincing and reliable method for improving relabeling accuracy. The proposed prediction-level and label-level consistency constraints seem also interesting and plausible.\n2.The combination of the proposed OT and curriculum learning for solving INL is also smooth and makes sense.", "weaknesses": "1.The effectiveness for alignment of global and local structures between samples and classes is not fully convincing. The ablation results for OT in Table 3 seem weak. The comparison between row (a) with 78.07 and row (b) with 78.65 suggests that the performance improvement brought by the proposed prediction-level and label-level constraints is limited. Moreover, the introduction of two additional constraints adds complexity to the optimization. Similarly, row (e), CSOT w/o Ω^{L}, achieves the best performance, indicating that the benefits brought by the prediction-level and label-level constraints are unstable.\n\n\n2.As the part that readers are most concerned about, the section 4.3, the loss function needs to be able to reflect the integrity of the method and the specific combination with its own innovation points. I can’t see the innovation of this article in the loss function here, and each loss term is an existing work. The work in this paper seems to be only used to build a dataset $\\mathcal{D}_{\\text{clean}}$ and  $\\mathcal{D}_{\\text{corrupted}}$ for training? This structure and method of writing can greatly weaken the contribution of this paper. Besides, I would like to suggest the authors provide an overall alghrothim to show the whole training process, where the proposed methods would have been used during each training epoch and the training objectives are not the key point in this paper.\n\n3.Considering the complexity of the proposed algorithm, and its marginal improvement over previous methods on two real datasets in Table 2, especially compared to NCE. The effectiveness of this work is questionable. Additional experiments are suggested, especially on Clothing1M. Besides, some related works[1] should be discussed which are published recently.  \n\n [1] OT-Filter: An Optimal Transport Filter for Learning with Noisy Labels (CVPR 2023)\n\n4.The results of row (g) in Table 3 are not sufficient to show that the performance improvement is brought about by the method in this paper. For example, we need a more detailed ablation study to explain the role of NCE loss and the role of CSOT.\n\n5.The structure of the article is confusing, which weakens the contribution of this article. Secondly, the introduction of some tool concepts is quite abrupt. For example, the proposal of Eq. (3) and (6) gives people a new form of OT formulation and will give specific solutions later. This expectation is affected by the entropy regularization in the following text, whose introduction is abrupt. This results in the final Eq. (16) which does not look significantly different from the original sinkhorn algorithm. It is recommended to introduce sinkhorn from the beginning and to emphasize that there are new constraints based on it.\n\n6.The characters in the figures in the experimental part are too small, which affects reading. At the same time, why are there two figures, and their labels are figure 2? Moreover, Fig. 4 is quoted in the description of the text, but there is no figure 4 in fact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback on our paper. Here we address your detailed comments below:\n\n> **Q1(1)**. The comparison between classical OT (row(a)) and Structure-aware OT (row(b)) in Table 3 suggests that the performance improvement brought by the proposed two regularization terms is limited. Row(e), CSOT w/o Ω^{L}, achieves the best performance, indicating that the benefits brought by two regularization terms are unstable.\n\n**A1(1)**. Firstly, **the unsurprising improvement of Structure-aware OT (SOT) results from the inherent characteristics of the Learning with Noisy Labels (LNL) task**, where the feature space and noisy labels exhibit ambiguity in the early training stage, leading to biased label generation.\n\n**To ensure reliable pseudo-labels for LNL task, we introduce curriculum constraints to SOT, i.e. our CSOT, where structure-aware regularization terms and curriculum constraints mutually enhance each other.** \n\nAs shown in Table3, our CSOT significantly boosts the classical OT (row(a)) performance from 78.07 to 81.85, achieving the best result. \n\nAs shown in Figure S6 (Appendix B.5), the CSOT design ensures the high accuracy of clean labels during the early training stage, which helps CSOT build a robust allocator in a high noise rate scenario.\n\n\n> **Q1(2)**. The introduction of two additional constraints adds complexity to the optimization.\n\n**A1(2)**. Our proposed efficient solver ensures minimal computational burden. Table 5 demonstrates our method's faster total training time compared to DivideMix and NCE.\n\n\n> **Q2(1)**. The loss function needs to be able to reflect the integrity of the method and the specific combination with its own innovation points. \n\n**A2(1)**. Firstly, we respectively disagree with this comment. We believe the loss function is not a necessary part to reflect the integrity and novelty of the method. In this paper, we do design a new objective function for OT, i.e. CSOT, incorporating of local coherent regularization terms and curriculum constraints. **Our proposed CSOT and the new solver instead of the loss function are the key innonvation of this paper**. Notably, our CSOT, aiming at identifying clean labels and correcting corrupted labels, is easy to be compatible with other Learning with Noisy Labels (LNL) losses. \n\nSecondly, we would like to claim that **modeling the sample selection and label correction [37][53] are also common directions for LNL as shown in the [R1], which clearly categorizes recent methods into four directions including \"Sample Selection\" and \"Robust Loss Design\"**.\n\n[R1] [TNNLS 2022] Learning from noisy labels with deep neural networks: A survey\n\n\n> **Q2(2)**. Suggest the authors provide an overall algorithm to show the whole training process.\n\n**A2(2)**. We provided the overall algorithm in Algorithm S3 (Appendix A.3).\n\n\n> **Q3(1)**. Additional experiments on Clothing1M are suggested.\n\n**A3(1)**. We conducted the experiment on the Clothing1M dataset in Table S7 (Appendix B.5).\n\n\n> **Q3(2)**. The related work [CVPR 2023] OT-Filter should be discussed.\n\n**A3(2)**. Thanks for your advice. We discuss this work in the following paragraph, and they will be involved in our final version.\n\nFirstly, we clarify that OT-Filter is a concurrent study, publicly accessible post the NeurIPS 2023 submission deadline.\n\nSecondly, our proposed CSOT methodology differs significantly from OT-Filter. OT-Filter involves standard OT-based pseudo-labeling with extra sparsity regularization, whereas CSOT concurrently considers both inter- and intra-distribution structures of samples to construct a robust curriculum allocator for denoising and relabeling.\n\nThirdly, CSOT's superiority over OT-Filter on CIFAR-10/100 and Clothing 1M dataset is demonstrated in Table G1, G2, G3 (global response pdf).\n\n\n> **Q4**. Need a more detailed ablation study to explain the role of NCE loss and the role of CSOT. \n\n**A4**. Firstly, **we did conduct detailed ablation study in Figure 2 (page 9)** with NCE and CSOT, comparing the performance of clean label identification and corrupted label correction. We will enlarge the font size in Figure 2 to make it more clear. **Despite using a similar learning objective as NCE, CSOT excels in sieving and relabeling, confirming our CSOT's effectiveness.** \n\nSecondly, our CSOT serving as a sample selection and pseudo-labeling step, akin to NCE. Thus, we adopt NCE's existing semi-supervised learning objectives, which is not a specially tailored LNL loss. \n\n\n> **Q5**. It is recommended to introduce sinkhorn from the beginning and to emphasize that there are new constraints based on it.\n\n**A5**. Thanks for your advice. We will move Line 196-200 (Section 5) to Section 3 Preliminaries to introduce Sinkhorn algorithm from the beginning in our final version.\n\n\n> **Q6**. Figure readability including mislabeled index and font size.\n\n**A6**. Thank you for advice. Figure 2 on page 9 is wrongly identified as Figure 4. We will revise these in our final version.\n\n\n> **Q7**. More detailed description of the curriculum constraint in Eq.(6). Why should $Q 1_C$ not exceed $\\frac{1}{B} 1_B$? Furthermore, it seems that $\\frac{m}{C} 1_{C}$ is not a simplex.\n\n**A7**. Firstly, the equality constraint $\\alpha=\\frac{1}{B}1_B$ in Eq.(2) signifies equal assignment budget for each sample, demanding uniform mapping intensity to the class centroid. **For the purpose of partial assignment, our CSOT relaxing this equality as $Q1_C\\leq\\frac{1}{B}1_B$.**\n\nSecondly, **while maintaining column equality ($Q^T1_B=\\frac{m}{C}1_C$), $m\\in[0,1]$ regulates total coupling sum to control the curriculum budget**, as in Line 123-124. We visually demonstrate CSOT's coupling matrix in Figure S4 (Appendix B.2), revealing the curriculum factor $m$ effect.\n\n\n> **Q8**. Suggest adding a description for the symbol ⌊·⌋ to clarify top-k selection in Eq.(7). \n\n**A8**. Thanks for your advice. We will revise this in our final version."}, {"review_id": "E7aOATIqMB", "reviewer": "Reviewer_7s92", "summary": "The paper studies the problem of noisy label learning. The paper adopts an optimal transport approach to generate pseudo labels for noisy samples. Particularly, the paper builds on the existing method and adds additional regularization terms to enforce the consistency between sample classes and learned representations. The paper also extends the sinkhorn algorithm to solve the proposed OT objective efficiently. Empirically, the proposed method has improved performance over baselines on widely used datasets with various noisy ratios.", "questions": "1. Please address the weaknesses 2 3 and 4.\n\n2. What's an explanation for the experimental results that the given method does better in top-1 ACC but worse in top-5 for the imagenet and webvision datasets?\n\n3. The global relationship modeled by OT depends on the batch size. How does the batch size affect the empirical performance? How does the number of total classes in the training set affect the choice of the ideal batch size for the OT step? ", "limitations": "Yes.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper extends the existing optimal transport approach to include the consistency between the sample representations and predictions/labels. It is a novel objective and a solid idea intuitively.\n\n2. The paper extends the sinkhorn algorithm to solve the proposed new OT objective efficiently.\n\n3. The proposed method has strong empirical performance, especially for high noise ratios.\n\n4. As a pseudo-labeling step, the proposed method can potentially work with other noisy label learning objectives.", "weaknesses": "1. The proposed method is a regularization of the existing optimal transport pseudo-labeling method. The novelty is thus limited.\n\n2. The introduced regularization uses the same weight kappa for the two terms, while the two terms could have quite different behaviors/values.\n\n3. It is not clear to me how does the method perform or should be modified in the case where the class distribution is imbalanced.\n\n4. The OT objective is not directly related to the training objective but serves as a sieving step. It would be great if a more comprehensive training objective can be formalized to include the OT based selection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and constructive suggestions on our paper. We address your detailed comments below:\n\n> **Q1**. the proposed method is a regularization of the existing OT pseudo-labeling (PL) method. The novelty is thus limited.\n\n**A1**. Firstly, we would like to justify that our CSOT formulation not only involves two local coherent regularization terms but also curriculum constraints. Notably, **CSOT introduces a novel OT formulation, necessitating a fresh solver. Thus, we devise an lightspeed computational method**.\n\nSecondly, we emphasize that directly applying the original OT to certain PL tasks may result in sub-optimal performance. Consequently, it becomes necessary for researchers to investigate more adaptive OT formulations for specific problems [9][54], including off-the-shelf variants like unbalanced OT and partial OT, to address the specific challenges of the problem. In this context, we propose a novel CSOT formulation tailored for the denoising and relabeling task.\n\nThirdly, we highlight that our work serves as a valuable example of developing a customized OT formulation and a corresponding solver. By showcasing our method's adaptability and efficacy, we aim to contribute to the wider application and exploration of OT across various domains.\n\n\n> **Q2**. The introduced regularization uses the same weight kappa for the two terms, while the two terms could have quite different behaviors/values.\n\n**A2**. We agree with the reviewer that there may exist better $\\kappa_1$ and $\\kappa_2$ for $\\Omega^P$ and $\\Omega^L$ accordingly in Eq.(3). However, practically, it is rather cumbersome to tune separate hyperparameters for different datasets or noise rates, and we set a unified $\\kappa$ for simplicity. \n\nMoreover, we find the $\\kappa_1$ and $\\kappa_2$ are not very sensitive as shown in the Table G4 (global response pdf). The table also reveals that in scenarios with high noise rates like CIFAR-100 sym-0.8, prioritizing the prediction-level term with a higher weight is more advantageous due to label unreliability.\n\n\n> **Q3**. How does the method perform or should be modified in the case where the class distribution is imbalanced?\n\n**A3**. Thank you for raising the concern. Following existing work [18][54], like SwAV, we adopt a uniform class marginal vector for simplicity. While we do not address the class imbalance in this paper within the scope of LNL, we believe our approach can be extended for this purpose in future work. Here's a possible solution: we can introduce an outer minimizer to optimize an appropriate imbalanced class distribution $\\beta$:\n$\\min_\\beta \\min_{Q\\in\\Pi^c(\\alpha,\\beta)}\n            \\left<C, Q\\right>\n        +\\kappa\\Omega(Q)\n        +\\varepsilon\\left<Q, \\log Q\\right>$\n\ns.t. $\\beta \\in $ {$ \\beta\\in\\mathbb{R}_{+}^{C} | \\sum_i^C \\beta_i=m $}.\n\n\n> **Q4**. The OT objective is not directly related to the training objective but serves as a sieving step. It would be great if a more comprehensive training objective can be formalized to include the OT-based selection.\n\n**A4**. Thank you for your constructive suggestion.\n\nFirstly, we claim that an additional OT-based training objective isn't essential. To explain this, our CSOT-based Pseudo-Labeling (PL) aims at selecting top-confident samples and relabeling them with reliable labels. Following existing work [18][54], reliable labels from OT-based PL seamlessly plug into diverse supervised or semi-supervised losses. Thus, CSOT-based denoising and relabeling, followed by training with standard objectives, offers flexibility.\n\nSecondly, our offline sieving optimization, unsuitable for model training. To be more specific, following DivideMix and NCE, our learning scheme is based on semi-supervised learning as shown in Algorithm S3 (Appendix A.3). And the sieving step based on CSOT has to be offline to get the labeled and unlabeled dataset for the follow-up semi-supervised learning step. Therefore, it is able to learn from the selected and relabeled samples based on off-the-shelf training objectives instead of the offline sieving objective.\n\n\n> **Q5**. The explanation for the experimental results that the given method does better in top-1 ACC but worse in top-5 for the Imagenet and WebVision datasets.\n\n**A5**. Thank you for raising the concern. There are two potential reasons behind this.\n\nFirstly, we reckon that the class-imbalance scenario in the Webvision dataset can not be fully well-addressed in our work. As mentioned in Section 7, class-imbalance cases are specifically not considered in our work and hence we simply adopt uniform distribution as the categories distribution. The uniform distribution helps CSOT allocate equal focus to long-tailed categories, favoring top-1 accuracy. However, CSOT may mislabel some major class samples into unrelated categories, hampering top-5 accuracy. Nevertheless, we believe our CSOT can be extended to class-imbalanced scenarios in future work.\n\nSecondly, the existing work NCE employs a co-training scheme with two models, which can well avoid memorizing the noise. Conversely, we solely train one model, which may lead to incorrect labels being memorized. \n\n\n> **Q6**. How does the batch size affect the empirical performance? How does the number of total classes in the training set affect the choice of the ideal batch size for the OT step?\n\n**A6**. Thank you for raising the concern. Firstly, we would like to claim that to fully capture the local and global structure of the data, more samples per class is better. However, a larger batch size enlarges the OT matrix, potentially affecting computational efficiency. To balance accuracy and efficiency, we guarantee each class has at least 20 samples and set a batch size of 2000 for offline denoising and relabeling. Additionally, we employ an accumulated memory bank to store each mini-batch sample to fill 2000, which avoids the GPU memory limit exceeded problem."}]

Author rebuttal:
We thank the reviewers for their careful reading of our paper and help with improving our manuscript. We sincerely appreciate that you find our work:
* proposes novel objective which is a solid idea (Reviewer 7s92)
* proposes a convincing, reliable, interesting and plausible method (Reviewer Zh5c)
* is strongly motivated by theoretical analysis (Reviewer ZdDn)
* provides a very detailed derivation for the lightspeed computational method (Reviewer gpRK)
* exhibits superior and strong performance (Reviewer ZdDn, Reviewer 7s92)
* introduces a new perspective and potentially brings fresh insights to the field (Reviewer gpRK)

We thank the reviewers for their careful reading of our paper and help with improving our manuscript. We sincerely appreciate that you find our work:
* proposes novel objective which is a solid idea (Reviewer 7s92)
* proposes a convincing, reliable, interesting and plausible method (Reviewer Zh5c)
* is strongly motivated by theoretical analysis (Reviewer ZdDn)
* provides a very detailed derivation for the lightspeed computational method (Reviewer gpRK)
* exhibits superior and strong performance (Reviewer ZdDn, Reviewer 7s92)
* introduces a new perspective and potentially brings fresh insights to the field (Reviewer gpRK)

In the subsequent sections, we aim to address the concerns and questions you raised, offering a comprehensive item-by-item response to each of your comments.

We have provided some additional experiments results as reviewers suggest. Due to space limitations, we've displayed the results table **in the global response PDF** for Reviewer Zh5c Q3(2) and Reviewer 7s92 Q2.

Meta-review: The reviewers found the paper interesting, with an OT approach for classification in the presence of label noise and good performances but had a few concerns regarding limited novelty of applying OT for PL and label, limited performance gains, missing references and baselines,  lack of sensitivity analysis, which lead to a borderline positive score.

The authors did a good rebuttal by providing new numerical experiments and clarification to the reviewers concerns. the rebuttal was appreciated and two of the most negative reviewers increased their score. The final decision is to accept the paper but strongly suggest that the authors take into account all the reviewers comments in the main paper and supplementary including discussing the missing references and implementing them (all the discussed OT for noise label methods) and providing variances on more than 3 realizations in the final paper. An effort should also be made to explain better the proposed regularization, in particular the relation of equation (3) with quadratic OT formulations (Fused Gromov-Wasserstein, Laplacian regularized OT (Ferradans 2014) ) whose optimizer was used.

List of methods discussed by reviewers that should be discussed and compared (OT-Filter is contemporary but the authors can be commended since they compared to it in the rebuttal):

Karim, N., Rizve, M. N., Rahnavard, N., Mian, A., & Shah, M., 2022, Unicon: Combating label noise through uniform selection and contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9676-9686).

Feng, C., Ren, Y., & Xie, X. , 2023, OT-Filter: An Optimal Transport Filter for Learning With Noisy Labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 16164-16174).

Damodaran, B.B., Flamary, R., Seguy, V. and Courty, N., 2020. An entropic optimal transport loss for learning deep neural networks under label noise in remote sensing images. Computer Vision and Image Understanding, 191, p.102863.
Decision: Accept (poster)