Title: CSOT: Curriculum and Structure-Aware Optimal Transport for Learning with Noisy Labels
Venue: NeurIPS 2023
Accepted: yes
Keywords: Learning with Noisy Labels

Optimal Transport

Curriculum Learning
Abstract: Learning with noisy labels (LNL) poses a significant challenge in training a well-generalized model while avoiding overfitting to corrupted labels.
Recent advances have achieved impressive performance by identifying clean labels and correcting corrupted labels for training.
However, the current approaches rely heavily on the model’s predictions and evaluate each sample independently without considering either the global or local structure of the sample distribution.
These limitations typically result in a suboptimal solution for the identification and correction processes, which eventually leads to models overfitting to incorrect labels.
In this paper, we propose a novel optimal transport (OT) formulation, called Curriculum and Structure-aware Optimal Transport (CSOT). CSOT concurrently considers the inter- and intra-distribution structure of the samples to construct a robust denoising and relabeling allocator.
During the training process, the allocator incrementally assigns reliable labels to a fraction of the samples with the highest confidence. 
These labels have both global discriminability and local coherence.
Notably, CSOT is a new OT formulation with a nonconvex objective function and curriculum constraints, so it is not directly compatible with classical OT solvers. 
Here, we develop a lightspeed computational method that involves a scaling iteration within a generalized conditional gradient framework to solve CSOT efficiently.
Extensive experiments demonstrate the superiority of our method over the current state-of-the-arts in LNL.

CSOT: Curriculum and Structure-Aware Optimal Transport for Learning with Noisy Labels

Abstract

Learning with noisy labels (LNL) poses a significant challenge in training a wellgeneralized model while avoiding overfitting to corrupted labels. Recent advances have achieved impressive performance by identifying clean labels and correcting corrupted labels for training. However, the current approaches rely heavily on the models predictions and evaluate each sample independently without considering either the global or local structure of the sample distribution. These limitations typically result in a suboptimal solution for the identification and correction processes, which eventually leads to models overfitting to incorrect labels. In this paper, we propose a novel optimal transport (OT) formulation, called Curriculum and Structure-aware Optimal Transport (CSOT). CSOT concurrently considers the inter- and intra-distribution structure of the samples to construct a robust denoising and relabeling allocator. During the training process, the allocator incrementally assigns reliable labels to a fraction of the samples with the highest confidence. These labels have both global discriminability and local coherence. Notably, CSOT is a new OT formulation with a nonconvex objective function and curriculum constraints, so it is not directly compatible with classical OT solvers. Here, we develop a lightspeed computational method that involves a scaling iteration within a generalized conditional gradient framework to solve CSOT efficiently. Extensive experiments demonstrate the superiority of our method over the current state-of-the-arts in LNL. Code is available at https://github.com/changwxx/CSOT-for-LNL .

1 Introduction

Deep neural networks (DNNs) have significantly boosted performance in various computer vision tasks, including image classification [33], object detection [61], and semantic segmentation [32]. However, the remarkable performance of deep learning algorithms heavily relies on large-scale highquality human annotations, which are extremely expensive and time-consuming to obtain. Alternatively, mining large-scale labeled data based on a web search and user tags [49, 37] can provide a cost-effective way to collect labels, but this approach inevitably introduces noisy labels. Since DNNs can so easily overfit to noisy labels [4, 79], such label noise can significantly degrade performance, giving rise to a challenging task: learning with noisy labels (LNL) [50, 52, 46].

Numerous strategies have been proposed to mitigate the negative impact of noisy labels, including loss correction based on transition matrix estimation [35], re-weighting [60], label correction [76]

and sample selection [52]. Recent advances have achieved impressive performance by identifying clean labels and correcting corrupted labels for training. However, current approaches rely heavily on the models predictions to identify or correct labels even if the model is not yet sufficiently trained. Moreover, these approaches often evaluate each sample independently, disregarding the global or local structure of the sample distribution. Hence, the identification and correction process results in a suboptimal solution which eventually leads to a model overfitting to incorrect labels.

In light of the limitations of distribution modeling, optimal transport (OT) offers a promising solution by optimizing the global distribution matching problem that searches for an efficient transport plan from one distribution to another. To date, OT has been applied in various machine learning tasks [11, 83, 28]. In particular, OT-based pseudo-labeling [11, 73] attempts to map samples to class centroids, while considering the inter-distribution matching of samples and classes. However, such an approach could also produce assignments that overlook the inherent coherence structure of the sample distribution, i.e. intra-distribution coherence. More specifically, the cost matrix in OT relies on pairwise metrics, so two nearby samples could be mapped to two far-away class centroids (Fig. 1).

In this paper, to enhance intra-distribution coherence, we propose a new OT formulation for denoising and relabeling, called Structure-aware Optimal Transport (SOT). This formulation fully considers the intra-distribution structure of the samples and produces robust assignments with both global discriminability and local coherence . Technically speaking, we introduce local coherent regularized terms to encourage both prediction- and label-level local consistency in the assignments. Furthermore, to avoid generating incorrect labels in the early stages of training or cases with high noise ratios, we devise Curriculum and Structure-aware Optimal Transport (CSOT) based on SOT. CSOT constructs a robust denoising and relabeling allocator by relaxing one of the equality constraints to allow only a fraction of the samples with the highest confidence to be selected. These samples are then assigned with reliable pseudo labels. The allocator progressively selects and relabels batches of high-confidence samples based on an increasing budget factor that controls the number of selected samples. Notably, CSOT is a new OT formulation with a nonconvex objective function and curriculum constraints, so it is significantly different from the classical OT formulations. Hence, to solve CSOT efficiently, we developed a lightspeed computational method that involves a scaling iteration within a generalized conditional gradient framework [59].

Our contribution can be summarized as follows: 1) We tackle the denoising and relabeling problem in LNL from a new perspective, i.e. simultaneously considering the interand intra-distribution structure for generating superior pseudo labels using optimal transport. 2) To fully consider the intrinsic coherence structure of sample distribution, we propose a novel optimal transport formulation, namely Curriculum and Structure-aware Optimal Transport (CSOT), which constructs a robust denoising and relabeling allocator that mitigates error accumulation. This allocator selects a fraction of high-confidence samples, which are then assigned reliable labels with both global discriminability and local coherence . 3) We further develop a lightspeed computational method that involves a scaling iteration within a generalized conditional gradient framework to efficiently solve CSOT. 4) Extensive experiments demonstrate the superiority of our method over state-of-the-art methods in LNL.

2 Related Work

Learning with noisy labels. LNLis a well-studied field with numerous strategies having been proposed to solve this challenging problem, such as robust loss design [82, 70], loss correction [35, 56], loss re-weighting [60, 80] and sample selection [52, 31, 41]. Currently, the methods that are delivering superior performance mainly involve learning from both selected clean labels and relabeled corrupted labels [46, 45]. The mainstream approaches for identifying clean labels typically rely on the small-loss criterion [31, 77, 71, 14]. These methods often model per-sample loss distributions using a Beta Mixture Model [51] or a Gaussian Mixture Model [57], treating samples with smaller loss as clean ones [3, 71, 46]. The label correction methods, such as PENCIL [76], Selfie [63], ELR [50], and DivideMix [46], typically adopt a pseudo-labeling strategy that leverages the DNNs predictions to correct the labels. However, these approaches evaluate each sample independently without considering the correlations among samples, which leads to a suboptimal identification and correction solution. To this end, some work [55, 45] attempt to leverage k -nearest neighbor predictions [6] for clean identification and label correction. Besides, to further select and correct noisy labels robustly,

OT Cleaner [73], as well as concurrent OT-Filter [23], designed to consider the global sample distribution by formulating pseudo-labeling as an optimal transport problem. In this paper, we propose CSOT to construct a robust denoising and relabeling allocator that simultaneously considers both the global and local structure of sample distribution so as to generate better pseudo labels.

Optimal transport-based pseudo-labeling. OT is a constrained optimization problem that aims to find the optimal coupling matrix to map one probability distribution to another while minimizing the total cost [40]. OT has been formulated as a pseudo-labeling (PL) technique for a range of machine learning tasks, including class-imbalanced learning [44, 28, 68], semi-supervised learning [65, 54, 44], clustering [5, 11, 25], domain adaptation [83, 12], label refinery [83, 68, 73, 23], and others. Unlike prediction-based PL [62], OT-based PL optimizes the mapping samples to class centroids, while considering the global structure of the sample distribution in terms of marginal constraints instead of per-sample predictions. For example, Self-labelling [5] and SwA V [11], which are designed for self-supervised learning, both seek an optimal equal-partition clustering to avoid the models collapse. In addition, because OT-based PL considers marginal constraints, it can also consider class distribution to solve class-imbalance problems [44, 28, 68]. However, these approaches only consider the inter-distribution matching of samples and classes but do not consider the intradistribution coherence structure of samples. By contrast, our proposed CSOT considers both the inter- and intra-distribution structure and generates superior pseudo labels for noise-robust learning.

Curriculum learning. Curriculum learning (CL) attempts to gradually increase the difficulty of the training samples, allowing the model to learn progressively from easier concepts to more complex ones [42]. CL has been applied to various machine learning tasks, including image classification [38, 84], and reinforcement learning [53, 2]. Recently, the combination of curriculum learning and pseudo-labeling has become popular in semi-supervised learning. These methods mainly focus on dynamic confident thresholding [69, 29, 75] instead of adopting a fixed threshold [62]. Flexmatch [78] designs class-wise thresholds and lowers the thresholds for classes that are more difficult to learn. Different from dynamic thresholding approaches, SLA [65] only assigns pseudo labels to easy samples gradually based on an OT-like problem. In the context of LNL, CurriculumNet [30] designs a curriculum by ranking the complexity of the data using its distribution density in a feature space. Alternatively, RoCL [85] selects easier samples considering both the dynamics of the per-sample loss and the output consistency. Our proposed CSOT constructs a robust denoising and relabeling allocator that gradually assigns high-quality labels to a fraction of the samples with the highest confidence. This encourages both global discriminability and local coherence in assignments.

3 Preliminaries

Optimal transport. Here we briefly recap the well-known formulation of OT. Given two probability simplex vectors α and β indicating two distributions, as well as a cost matrix C ∈ R | α |×| β | , where | α | denotes the dimension of α , OT aims to seek the optimal coupling matrix Q by minimizing the following objective

where ⟨· , ·⟩ denote Frobenius dot-product. The coupling matrix Q satisfies the polytope Π ( α , β ) = { Q ∈ R | α |×| β | + | Q 1 | β | = α , Q ⊤ 1 | α | = β } , where α and β are essentially marginal probability vectors. Intuitively speaking, these two marginal probability vectors can be interpreted as coupling budgets, which control the mapping intensity of each row and column in Q .

Pseudo-labeling based on optimal transport. Let P ∈ R B × C + denote classifier softmax predictions, where B is the batch size of samples, and C is the number of classes. The OT-based PL considers mapping samples to class centroids and the cost matrix C can be formulated as -log P [65, 68]. We can rewrite the objective for OT-based PL based on Problem (1) as follows

where 1 d indicates a d -dimensional vector of ones. The pseudo-labeling matrix can be obtained by normalization: B Q . Unlike prediction-based PL [62] which evaluates each sample independently,

Localglyph[suppress]Prediction

Consistency

Localglyph[suppress]Labelglyph[suppress]

Consistency

ClassicalψOT

Structure

↩ awareψOTψ↼Ours↽

Cleanψsamples

Corruptedψsamples

Decisionψboundary

Implicitψclassψcentroids

Selectedψandψlabeled

口

t

∓

I

©

L

∗glyph[intersectionsq]∗∇∗∓

∓

I

©

L

∗glyph[supersetsqequal]∗∇∗∓

∓

I

©

L

∗∫∗∇∗∇∗∓

OT-based PL considers inter-distribution matching of samples and classes, as well as the global structure of sample distribution, thanks to the equality constraints.

Sinkhorn algorithm for classical optimal transport problem. Directly optimizing the exact OT problem would be time-consuming, and an entropic regularization term is introduced [19]: min Q ∈ Π ( α , β ) ⟨ C , Q ⟩ + ε ⟨ Q , log Q ⟩ , where ε > 0 . The entropic regularization term enables OT to be approximated efficiently by the Sinkhorn algorithm [19], which involves matrix scaling iterations executed efficiently by matrix multiplication on GPU.

4 Methodology

Problem setup. Let D train = { ( x i , y i ) } N i =1 denote the noisy training set, where x i is an image with its associated label y i over C classes, but whether the given label is accurate or not is unknown. We call the correctly-labeled ones as clean , and the mislabeled ones as corrupted . LNL aims to train a network that is robust to corrupted labels and achieves high accuracy on a clean test set.

4.1 Structure-Aware Optimal Transport for Denoising and Relabeling

Even though existing OT-based PL considers the global structure of sample distribution, the intrinsic coherence structure of the samples is ignored. Specifically, the cost matrix in OT relies on pairwise metrics and thus two nearby samples could be mapped to two far-away class centroids. To further consider the intrinsic coherence structure, we propose a Structure-aware Optimal Transport (SOT) for denoising and relabeling, which promotes local consensus assignment by encouraging predictionlevel and label-level consistency, as shown in Fig. 1.

Our proposed SOT for denoising and relabeling is formulated by adding two local coherent regularized terms based on Problem (2). Given a cosine similarity S ∈ R B × B among samples in feature space, a one-hot label matrix L ∈ R B × C transformed from given noisy labels, and a softmax prediction matrix P ∈ R B × C + , SOT is formulated as follows

where the local coherent regularized terms Ω P and Ω L encourages prediction-level and label-level local consistency respectively, and are defined as follows

where ⊙ indicates element-wise multiplication. To be more specific, Ω P encourages assigning larger weight to Q ik and Q jk if the i -th sample is very close to the j -th sample, and their predictions P ik and P jk from the k -th class centroid are simultaneously high. Analogously, Ω L encourages assigning larger weight to those samples whose neighborhood label consistency is rather high. Unlike the formulation proposed in [1, 16], which focuses on sample-to-sample mapping, our method introduces a sample-to-class mapping that leverages the intrinsic coherence structure within the samples.

4.2 Curriculum and Structure-Aware Optimal Transport for Denoising and Relabeling

In the early stages of training or in scenarios with a high noise ratio, the predictions and feature representation would be vague and thus lead to the wrong assignments for SOT. For the purpose of robust clean label identification and corrupted label correction, we further propose a Curriculum and Structure-aware Optimal Transport (CSOT), which constructs a robust curriculum allocator. This curriculum allocator gradually selects a fraction of the samples with high confidence from the noisy training set, controlled by a budget factor, then assigns reliable pseudo labels for them.

Our proposed CSOT for denoising and relabeling is formulated by introducing new curriculum constraints based on SOT in Problem (3). Given curriculum budget factor m ∈ [0 , 1] , our CSOT seeks optimal coupling matrix Q by minimizing following objective

Unlike SOT, which enforces an equality constraint on the samples, CSOT relaxes this constraint and defines the total coupling budget as m ∈ [0 , 1] , where m represents the expected total sum of Q . Intuitively speaking, m = 0 . 5 indicates that top 50% confident samples are selected from all the classes, avoiding only selecting the same class for all the samples within a mini-batch. And the budget m progressively increases during the training process, as shown in Fig. 1.

Based on the optimal coupling matrix Q solved from Problem (6), we can obtain pseudo label by argmax operation, i.e. ˆ y i = argmax j Q ij . In addition, we define the general confident scores of samples as W = { w 0 , w 1 , · · · , w B -1 } , where w i = Q i ˆ y i / ( m/C ) . Since our curriculum allocator assigns weight to only a fraction of samples controlled by m , we use topK ( S , k ) operation (return topk indices of input set S ) to identify selected samples denoted as δ i

where ⌊·⌋ indicates the round down operator. Then the noisy dataset D train can be splited into D clean and D corrupted as follows

4.3 Training Objectives

To avoid error accumulation in the early stage of training, we adopt a two-stage training scheme. In the first stage, the model is supervised by progressively selected clean labels and self-supervised by unselected samples. In the second stage, the model is semi-supervised by all denoised labels. Notably, we construct our training objective mainly based on Mixup loss L mix and Label consistency

̸

loss L lab same as NCE [45], and a self-supervised loss L simsiam proposed in SimSiam [15]. The detailed formulations of mentioned loss and training process are given in Appendix. Our two-stage training objective can be constructed as follows

5 Lightspeed Computation for CSOT

The proposed CSOT is a new OT formulation with nonconvex objective function and curriculum constraints, which cannot be solved directly by classical OT solvers. To this end, we develop a lightspeed computational method that involves a scaling iteration within a generalized conditional gradient framework to solve CSOT efficiently. Specifically, we first introduce an efficient scaling iteration for solving the OT problem with curriculum constraints without considering the local coherent regularized terms, i.e. Curriculum OT (COT). Then, we extend our approach to solve the proposed CSOT problem, which involves a nonconvex objective function and curriculum constraints.

5.1 Solving Curriculum Optimal Transport

For convenience, we formulate curriculum constraints in Probelm (6) in a more general form. Given two vectors α and β that satisfy ∥ α ∥ 1 ≥ ∥ β ∥ 1 = m , a general polytope of curriculum constraints Π c ( α , β ) is formulated as

For the efficient computation purpose, we consider an entropic regularized version of COT

where we denote the cost matrix C := -log P in Probelm (6) for simplicity. Inspired by [8], Problem (12) can be easily re-written as the Kullback-Leibler (KL) projection: min Q ∈ Π c ( α , β ) ε KL ( Q | e -C /ε ) . Besides, the polytope Π c ( α , β ) can be expressed as an intersection of two convex but not affine sets, i.e.

In light of this, Problem (12) can be solved by performing iterative KL projection between C 1 and C 2 , namely Dykstra's algorithm [21] shown in Appendix.

Lemma 1. (Efficient scaling iteration for Curriculum OT) When solving Problem (12) by iterating Dykstra's algorithm, the matrix Q ( n ) at n iteration is a diagonal scaling of K := e -C /ε , which is the element-wise exponential matrix of -C /ε :

where the vectors u ( n ) ∈ R | α | , v ( n ) ∈ R | β | satisfy v (0) = 1 | β | and follow the recursion formula

The proof is given in the Appendix. Lemma 1 allows a fast implementation of Dykstra's algorithm by only performing matrix-vector multiplications. This scaling iteration for entropic regularized COT is very similar to the widely-used and efficient Sinkhorn Algorithm [19], as shown in Algorithm 1.

5.2 Solving Curriculum and Structure-Aware Optimal Transport

In the following, we propose to solve CSOT within a Generalized Conditional Gradient (GCG) algorithm [59] framework, which strongly relies on computing Curriculum OT by scaling iterations

Algorithm 1 Efficient scaling iteration for entropic regularized Curriculum OT

1: Input: Cost matrix C , marginal constraints vectors α and β , entropic regularization weight ε

2: Initialize: K ← e -C /ε , v (0) ← 1 | β |

4: for n = 1 , 2 , 3 , . . . do

3: Compute: K α ← K diag ( α ) 1 | α |×| β | , K ⊤ β ← K ⊤ diag ( β ) 1 | β |×| α | // Saving computation

5: u ( n ) ← min ( 1 | α | K α v ( n -1) , 1 | α | )

6: v ( n ) ← 1 | β | K ⊤ β u ( n )

7: end for

8: Return: diag ( u ( n ) ) K diag ( v ( n )

)

in Algorithm 1. The conditional gradient algorithm [27, 36] has been used for some penalized OT problems [24, 17] or nonconvex Gromov-Wasserstein distances [58, 67, 13], which can be used to solve Problem (3) directly.

For simplicity, we denote the local coherent regularized terms as Ω( · ) := Ω P ( · ) + Ω L ( · ) , and give an entropic regularized CSOT formulation as follows:

Since the local coherent regularized term Ω P ( · ) is differentiable, Problem (16) can be solved within the GCG algorithm framework, shown in Algorithm 2. And the linearization procedure in Line 5 can be computed efficiently by the scaling iteration proposed in Sec 5.1.

Algorithm 2 Generalized conditional gradient algorithm for entropic regularized CSOT

1: Input: Cost matrix C , marginal constraints vectors α and β , entropic regularization weight ε , local coherent regularization weight κ , local coherent regularization function Ω : R | α |×| β | → R , and its gradient function ∇ Ω : R | α |×| β | → R | α |×| β |

3: for i = 1 , 2 , 3 , . . . do

2: Initialize: Q (0) ← αβ T

4: G ( i ) ← Q ( i ) + κ ∇ Ω( Q ( i ) ) // Gradient computation

6: Choose η ( i ) ∈ [0 , 1] so that it satisfies the Armijo rule // Backtracking line-search

5: ˜ Q ( i ) ← argmin Q ∈ Π c ( α , β ) 〈 Q , G ( i ) 〉 + ε ⟨ Q , log Q ⟩ // Linearization, solved efficiently by Algorithm 1

7: Q ( i +1) ← ( 1 -η ( i ) ) Q ( i ) + η ( i ) ˜ Q ( i ) // Update 8: end for 9: Return: Q ( i )

6 Experiments

6.1 Implementation Details

We conduct experiments on three standard LNL benchmark datasets: CIFAR-10 [43], CIFAR-100 [43] and Webvision [49]. We follow most implementation details from the previous work DivideMix [46] and NCE [45]. Here we provide some specific details of our approach. The warm-up epochs are set to 10/30/10 for CIFAR-10/100/Webvision respectively. For CIFAR-10/100, the supervised learning epoch T sup is set to 250 , and the semi-supervised learning epoch T semi is set to 200 . For Webvision, T sup = 80 and T semi = 70 . For all experiments, we set λ 1 = 1 , λ 2 = 1 , ε = 0 . 1 , κ = 1 . And we adopt a simple linear ramp for curriculum budget, i.e. m = min(1 . 0 , m 0 + t -1 T sup -1 ) with an initial budget m 0 = 0 . 3 . For the GCG algorithm, the number of outer loops is set to 10, and the number for inner scaling iteration is set to 100. The batch size B for denoising and relabeling is set to 1024 . More details will be provided in Appendix.

6.2 Comparison with the State-of-the-Arts

Synthetic noisy datasets. Our method is validated on two synthetic noisy datasets, i.e. CIFAR10 [43] and CIFAR-100 [43]. Following [46, 45], we conduct experiments with two types of label noise: symmetric and asymmetric . Symmetric noise is injected by randomly selecting a percentage of samples and replacing their labels with random labels. Asymmetric noise is designed to mimic the pattern of real-world label errors, i.e. labels are only changed to similar classes ( e.g. cat ↔ dog). As shown in Tab. 1, our CSOT has surpassed all the state-of-the-art works across most of the noise ratios. In particular, our CSOT outperforms the previous state-of-the-art method NCE [45] by 2 . 3% , 3 . 1% and 9 . 4% under a high noise rate of CIFAR-10 sym-0.8, CIFAR-100 sym-0.8/0.9, respectively.

Real-world noisy datasets. Additionally, we conduct experiments on a large-scale dataset with real-world noisy labels, i.e. WebVision [49]. WebVision contains 2.4 million images crawled from the web using the 1,000 concepts in ImageNet ILSVRC12 [20]. Following previous works [46, 45], we conduct experiments only using the first 50 classes of the Google image subset for a total of ∼ 61,000 images. As shown in Tab. 2, our CSOT surpasses other methods in top-1 accuracy on both Webvision and ILSVRC12 validation sets, demonstrating its superior performance in dealing with real-world noisy datasets. Even though NCE achieves better top-5 accuracy, it suffers from high time costs (using a single NVIDIA A100 GPU) due to the co-training scheme, as shown in Tab. S5.

6.3 Ablation Studies and Analysis

Effectiveness of CSOT-based denoising and relabeling. To verify the effectiveness of each component in our CSOT, we conduct comprehensive ablation experiments, shown in Tab. 4. Compared to classical OT, Structure-aware OT, and Curriculum OT, our proposed CSOT has achieved superior performance. Specifically, our proposed local coherent regularized terms Ω P and Ω Q indeed contribute to CSOT, as demonstrated in Tab 4 (c)(d)(e). Furthermore, our proposed curriculum constraints yield an improvement of approximately 2% for both classical OT and structure-aware OT, as shown in Tab 4 (a)(b)(c). Particularly, under high noise ratios, the improvement can reach up to 4% , which demonstrates the effectiveness of the curriculum relabeling scheme.

Effectiveness of clean labels identification via CSOT. As shown in Tab. 4 (f), replacing our CSOT-based denoising and relabeling with GMM [46] for clean label identification significantly degrades the model performance. This phenomenon can be explained by the clean accuracy during training (Fig. 2a) and clean recall rate (Fig. 2c), in which our CSOT consistently outperforms other methods in accurately retrieving clean labels, leading to significant performance improvements. These experiments fully show that our CSOT can maintain both high quantity and high quality of clean labels during training.

Effectiveness of corrupted labels correction via CSOT. As shown in Tab. 4 (h), only training with identified clean labels leads to inferior model performance. Furthermore, replacing our CSOTbased denoising and relabeling with confidence thresholding (CT) [62] for corrupted label correction also degrades the model performance, as shown in Tab. 4 (i). The CT methods assign pseudo labels to samples based on model prediction, which is unreliable in the early training stage, especially under high noise rates. Our CSOT-based denoising and relabeling fully consider the inter- and intradistribution structure of samples, yielding more robust labels. Particularly, our CSOT outperforms NCE and DivideMix significantly in label correction, as demonstrated by the superior corrected accuracy in Fig. 2b and the improved clarity of the confusion matrix in Fig. S7.

Effectiveness of curriculum training scheme. According to the progressive clean and corrupted accuracy during the training process shown in Fig. 2a and Fig. 2b, our curriculum identification scheme ensures high accuracy in the early training stage, avoiding overfitting to wrong corrected labels. Note that since our model is trained using only a fraction of clean samples, it is crucial to employ a powerful supervised learning loss to facilitate better learning. Otherwise, the performance will be poor without the utilization of a powerful supervised training loss, as evidenced in Tab. 4 (g). In addition, the incorporation of self-supervised loss enhances noise-robust representation, particularly in high noise rate scenarios, as demonstrated in our experiments in Tab. 4 (j).

Time cost discussion for solving CSOT To verify the efficiency of our proposed lightspeed scaling iteration, we conduct some experiments for solving CSOT optimization problem of different input sizes on a single GPU NVIDIA A100. As demonstrated in Tab. 3, our proposed lightspeed

20%

40%

60%

80%

100%

Training progress

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Accuracy of selected clean labels

DivideMix

NCE

Ours

CIFAR10-sym90%

CIFAR100-sym50%

(a) Clean accuracy

20%

40%

60%

80%

100%

Training progress

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Accuracy of corrected labels

DivideMix

NCE

Ours

CIFAR10-sym90%

CIFAR100-sym50%

(b) Corrected accuracy

CIFAR10-sym20%

CIFAR10-asym40%

CIFAR100-sym90%

75

80

85

90

95

100

Recall rate (%)

92.0

85.4

77.5

96.7

97.6

79.9

97.6

98.2

82.8

Recall Rate of Clean Samples

DivideMix

NCE

Ours

(c) Clean recall rate

computational method that involves an efficient scaling iteration (Algorithm 1) achieves lower time cost compared to vanilla Dykstras algorithm (Algorithm S6). Specifically, compared to the vanilla Dykstra-based approach, our efficient scaling iteration version can achieve a speedup of up to 3.7 times, thanks to efficient matrix-vector multiplication instead of matrix-matrix multiplication. Moreover, even for very large input sizes, the computational time cost does not increase significantly.

7 Conclusion and Limitation

In this paper, we proposed Curriculum and Structure-aware Optimal Transport (CSOT), a novel solution to construct robust denoising and relabeling allocator that simultaneously considers the interand intra-distribution structure of samples. Unlike current approaches, which rely solely on the model's predictions, CSOT considers the global and local structure of the sample distribution to construct a robust denoising and relabeling allocator. During the training process, the allocator assigns reliable labels to a fraction of the samples with high confidence, ensuring both global discriminability and local coherence. To efficiently solve CSOT, we developed a lightspeed computational method that involves a scaling iteration within a generalized conditional gradient framework. Extensive experiments on three benchmark datasets validate the efficacy of our proposed method. While class-imbalance cases are not considered in this paper within the context of LNL, we believe that our approach can be further extended for this purpose.

References

David Alvarez-Melis, Tommi Jaakkola, and Stefanie Jegelka. Structured optimal transport. In International conference on artificial intelligence and statistics , pages 1771-1780. PMLR, 2018.

Shuang Ao, Tianyi Zhou, Guodong Long, Qinghua Lu, Liming Zhu, and Jing Jiang. Co-pilot: Collaborative planning and reinforcement learning on sub-task curriculum. Advances in Neural Information Processing Systems , 34:10444-10456, 2021.

Eric Arazo, Diego Ortego, Paul Albert, Noel OConnor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In International conference on machine learning , pages 312-321. PMLR, 2019.

Devansh Arpit, Stanisław Jastrz˛ ebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning , pages 233-242. PMLR, 2017.

YM Asano, C Rupprecht, and A Vedaldi. Self-labelling via simultaneous clustering and representation learning. In International Conference on Learning Representations , 2019.

Dara Bahri, Heinrich Jiang, and Maya Gupta. Deep k-nn for noisy labels. In International Conference on Machine Learning , pages 540-550. PMLR, 2020.

Heinz H Bauschke and Adrian S Lewis. Dykstras algorithm with bregman projections: A convergence proof. Optimization , 48(4):409-427, 2000.

Jean-David Benamou, Guillaume Carlier, Marco Cuturi, Luca Nenna, and Gabriel Peyré. Iterative bregman projections for regularized transportation problems. SIAM Journal on Scientific Computing , 37(2):A1111-A1138, 2015.

Dimitri P Bertsekas. Nonlinear programming. Journal of the Operational Research Society , 48(3):334334, 1997.

Lev M Bregman. The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR computational mathematics and mathematical physics , 7(3):200-217, 1967.

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems , 33:9912-9924, 2020.

Wanxing Chang, Ye Shi, Hoang Tuan, and Jingya Wang. Unified optimal transport framework for universal domain adaptation. Advances in Neural Information Processing Systems , 35:29512-29524, 2022.

Laetitia Chapel, Mokhtar Z Alaya, and Gilles Gasso. Partial optimal tranport with applications on positiveunlabeled learning. Advances in Neural Information Processing Systems , 33:2903-2913, 2020.

Pengfei Chen, Ben Ben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In International Conference on Machine Learning , pages 1062-1070. PMLR, 2019.

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 15750-15758, 2021.

Ching-Yao Chuang, Stefanie Jegelka, and David Alvarez-Melis. Infoot: Information maximizing optimal transport. In International Conference on Machine Learning , pages 6228-6242. PMLR, 2023.

Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamonjy. Optimal transport for domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 39:1853-1865, 2017.

Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 113-123, 2019.

Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems , 26:2292-2300, 2013.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Richard L Dykstra. An algorithm for restricted least squares regression. Journal of the American Statistical Association , 78(384):837-842, 1983.

Erik Englesson and Hossein Azizpour. Consistency regularization can improve robustness to label noise. In International Conference on Machine Learning Workshops, 2021 Workshop on Uncertainty and Robustness in Deep Learning , 2021.

Chuanwen Feng, Yilong Ren, and Xike Xie. Ot-filter: An optimal transport filter for learning with noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16164-16174, 2023.

Sira Ferradans, Nicolas Papadakis, Gabriel Peyré, and Jean-François Aujol. Regularized discrete optimal transport. SIAM Journal on Imaging Sciences , 7(3):1853-1882, 2014.

Enrico Fini, Enver Sangineto, Stéphane Lathuilière, Zhun Zhong, Moin Nabi, and Elisa Ricci. A unified objective for novel class discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9284-9292, 2021.

Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, Ievgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python optimal transport. Journal of Machine Learning Research , 22(78):1-8, 2021.

Marguerite Frank and Philip Wolfe. An algorithm for quadratic programming. Naval research logistics quarterly , 3(1-2):95-110, 1956.

Dandan Guo, Zhuo Li, He Zhao, Mingyuan Zhou, Hongyuan Zha, et al. Learning to re-weight examples with optimal transport for imbalanced classification. Advances in Neural Information Processing Systems , 35:25517-25530, 2022.

Lan-Zhe Guo and Yu-Feng Li. Class-imbalanced semi-supervised learning with adaptive thresholding. In International Conference on Machine Learning , pages 8082-8094. PMLR, 2022.

Sheng Guo, Weilin Huang, Haozhi Zhang, Chenfan Zhuang, Dengke Dong, Matthew R Scott, and Dinglong Huang. Curriculumnet: Weakly supervised learning from large-scale web images. In Proceedings of the European conference on computer vision , pages 135-150, 2018.

BoHan, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems , 31, 2018.

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision , pages 2961-2969, 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision , pages 630-645. Springer, 2016.

Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. Advances in neural information processing systems , 31, 2018.

Martin Jaggi. Revisiting frank-wolfe: Projection-free sparse convex optimization. In International conference on machine learning , pages 427-435. PMLR, 2013.

Lu Jiang, Di Huang, Mason Liu, and Weilong Yang. Beyond synthetic noise: Deep learning on controlled noisy labels. In International conference on machine learning , pages 4804-4815. PMLR, 2020.

Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander Hauptmann. Self-paced curriculum learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 29, 2015.

Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International conference on machine learning , pages 2304-2313. PMLR, 2018.

Leonid V Kantorovich. On the translocation of masses. Journal of mathematical sciences , 133(4):13811382, 2006.

Nazmul Karim, Mamshad Nayeem Rizve, Nazanin Rahnavard, Ajmal Mian, and Mubarak Shah. Unicon: Combating label noise through uniform selection and contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9676-9686, 2022.

Faisal Khan, Bilge Mutlu, and Jerry Zhu. How do humans teach: On curriculum learning and teaching dimension. Advances in neural information processing systems , 24, 2011.

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.

Zhengfeng Lai, Chao Wang, Sen-ching Cheung, and Chen-Nee Chuah. Sar: Self-adaptive refinement on pseudo labels for multiclass-imbalanced semi-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4091-4100, 2022.

Jichang Li, Guanbin Li, Feng Liu, and Yizhou Yu. Neighborhood collective estimation for noisy label identification and correction. In European Conference on Computer Vision , pages 128-145. Springer, 2022.

Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semisupervised learning. In International Conference on Learning Representations , 2019.

Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S Kankanhalli. Learning to learn from noisy labeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5051-5059, 2019.

Junnan Li, Caiming Xiong, and Steven CH Hoi. Learning from noisy data with robust representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 94859494, 2021.

Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data. CoRR , 2017.

Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda. Early-learning regularization prevents memorization of noisy labels. Advances in neural information processing systems , 33:20331-20342, 2020.

Zhanyu Ma and Arne Leijon. Bayesian estimation of beta mixture models with variational inference. IEEE Transactions on Pattern Analysis and Machine Intelligence , 33(11):2160-2173, 2011.

Eran Malach and Shai Shalev-Shwartz. Decoupling" when to update" from" how to update". Advances in neural information processing systems , 30, 2017.

Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. The Journal of Machine Learning Research , 21(1):7382-7431, 2020.

Vu Nguyen, Sachin Farfade, and Anton van den Hengel. Confident sinkhorn allocation for pseudolabeling. arXiv preprint arXiv:2206.05880 , 2022.

Diego Ortego, Eric Arazo, Paul Albert, Noel E O'Connor, and Kevin McGuinness. Multi-objective interpolation training for robustness to label noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6606-6615, 2021.

Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1944-1952, 2017.

Haim Permuter, Joseph Francos, and Ian Jermyn. A study of gaussian mixture models of color and texture features for image classification and segmentation. Pattern recognition , 39(4):695-706, 2006.

Gabriel Peyré, Marco Cuturi, and Justin Solomon. Gromov-wasserstein averaging of kernel and distance matrices. In International conference on machine learning , pages 2664-2672. PMLR, 2016.

Alain Rakotomamonjy, Rémi Flamary, and Nicolas Courty. Generalized conditional gradient: analysis of convergence and applications. arXiv preprint arXiv:1510.06567 , 2015.

Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International conference on machine learning , pages 4334-4343. PMLR, 2018.

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems , 28, 2015.

Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems , 33:596-608, 2020.

Hwanjun Song, Minseok Kim, and Jae-Gil Lee. Selfie: Refurbishing unclean samples for robust deep learning. In International Conference on Machine Learning , pages 5907-5915. PMLR, 2019.

Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander Alemi. Inception-v4, inceptionresnet and the impact of residual connections on learning. In Proceedings of the AAAI conference on artificial intelligence , volume 31, 2017.

Kai Sheng Tai, Peter D Bailis, and Gregory Valiant. Sinkhorn label allocation: Semi-supervised classification via annealed self-training. In International Conference on Machine Learning , pages 10065-10075. PMLR, 2021.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Titouan Vayer, Laetitia Chapel, Rémi Flamary, Romain Tavenard, and Nicolas Courty. Fused gromovwasserstein distance for structured objects. Algorithms , 13(9):212, 2020.

Haobo Wang, Mingxuan Xia, Yixuan Li, Yuren Mao, Lei Feng, Gang Chen, and Junbo Zhao. Solar: Sinkhorn label refinery for imbalanced partial-label learning. Advances in Neural Information Processing Systems , 35:8104-8117, 2022.

Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, Zhen Wu, and Jindong Wang. Freematch: Self-adaptive thresholding for semi-supervised learning. International Conference on Learning Representations , 2022.

Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 322-330, 2019.

Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 13726-13735, 2020.

Zhi-Fan Wu, Tong Wei, Jianwen Jiang, Chaojie Mao, Mingqian Tang, and Yu-Feng Li. Ngc: A unified framework for learning with open-world noisy data. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 62-71, 2021.

Jun Xia, Cheng Tan, Lirong Wu, Yongjie Xu, and Stan Z Li. Ot cleaner: Label correction as optimal transport. In IEEE International Conference on Acoustics, Speech and Signal Processing , pages 39533957. IEEE, 2022.

Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2691-2699, 2015.

Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semisupervised learning with dynamic thresholding. In International Conference on Machine Learning , pages 11525-11536. PMLR, 2021.

Kun Yi and Jianxin Wu. Probabilistic end-to-end noise correction for learning with noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 70177025, 2019.

Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In International Conference on Machine Learning , pages 7164-7173. PMLR, 2019.

Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems , 34:18408-18419, 2021.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3):107-115, 2021.

HaiYang Zhang, XiMing Xing, and Liang Liu. Dualgraph: A graph-based method for reasoning about label noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9654-9663, 2021.

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. International Conference on Learning Representations , 2017.

Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems , 31, 2018.

Kecheng Zheng, Wu Liu, Lingxiao He, Tao Mei, Jiebo Luo, and Zheng-Jun Zha. Group-aware label transfer for domain adaptive person re-identification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5310-5319, 2021.

Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Curriculum learning by optimizing learning dynamics. In International Conference on Artificial Intelligence and Statistics , pages 433-441. PMLR, 2021.

Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Robust curriculum learning: from clean label detection to noisy label self-correction. In International Conference on Learning Representations , 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of noisy label learning. The paper adopts an optimal transport approach to generate pseudo labels for noisy samples. Particularly, the paper builds on the existing method and adds additional regularization terms to enforce the consistency between sample classes and learned representations. The paper also extends the sinkhorn algorithm to solve the proposed OT objective efficiently. Empirically, the proposed method has improved performance over baselines on widely used datasets with various noisy ratios.\n\nSTRENGTHS:\n1. The paper extends the existing optimal transport approach to include the consistency between the sample representations and predictions/labels. It is a novel objective and a solid idea intuitively.\n\n2. The paper extends the sinkhorn algorithm to solve the proposed new OT objective efficiently.\n\n3. The proposed method has strong empirical performance, especially for high noise ratios.\n\n4. As a pseudo-labeling step, the proposed method can potentially work with other noisy label learning objectives.\n\nWEAKNESSES:\n1. The proposed method is a regularization of the existing optimal transport pseudo-labeling method. The novelty is thus limited.\n\n2. The introduced regularization uses the same weight kappa for the two terms, while the two terms could have quite different behaviors/values.\n\n3. It is not clear to me how does the method perform or should be modified in the case where the class distribution is imbalanced.\n\n4. The OT objective is not directly related to the training objective but serves as a sieving step. It would be great if a more comprehensive training objective can be formalized to include the OT based selection.\n\nQUESTIONS:\n1. Please address the weaknesses 2 3 and 4.\n\n2. What's an explanation for the experimental results that the given method does better in top-1 ACC but worse in top-5 for the imagenet and webvision datasets?\n\n3. The global relationship modeled by OT depends on the batch size. How does the batch size affect the empirical performance? How does the number of total classes in the training set affect the choice of the ideal batch size for the OT step?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel formulation of Optimal Transport (OT), named Curriculum and Structure-Aware Optimal Transport, for generating pseudo labels by considering both inter- and intra-distribution structures of samples. Moreover, to efficiently estimate the distribution's structure, the authors adopt a curriculum paradigm to progressively train the proposed denoising and relabeling allocator. Additionally, they present a computation method for the proposed CSOT that ensures faster processing speeds, reducing computational overhead. Experimentally, this paper achieved SOTA performance on various benchmarks.\n\nSTRENGTHS:\n1.Estimating the intra- and inter-structure coherence of samples is a convincing and reliable method for improving relabeling accuracy. The proposed prediction-level and label-level consistency constraints seem also interesting and plausible.\n2.The combination of the proposed OT and curriculum learning for solving INL is also smooth and makes sense.\n\nWEAKNESSES:\n1.The effectiveness for alignment of global and local structures between samples and classes is not fully convincing. The ablation results for OT in Table 3 seem weak. The comparison between row (a) with 78.07 and row (b) with 78.65 suggests that the performance improvement brought by the proposed prediction-level and label-level constraints is limited. Moreover, the introduction of two additional constraints adds complexity to the optimization. Similarly, row (e), CSOT w/o Ω^{L}, achieves the best performance, indicating that the benefits brought by the prediction-level and label-level constraints are unstable.\n\n\n2.As the part that readers are most concerned about, the section 4.3, the loss function needs to be able to reflect the integrity of the method and the specific combination with its own innovation points. I can’t see the innovation of this article in the loss function here, and each loss term is an existing work. The work in this paper seems to be only used to build a dataset $\\mathcal{D}_{\\text{clean}}$ and  $\\mathcal{D}_{\\text{corrupted}}$ for training? This structure and method of writing can greatly weaken the contribution of this paper. Besides, I would like to suggest the authors provide an overall alghrothim to show the whole training process, where the proposed methods would have been used during each training epoch and the training objectives are not the key point in this paper.\n\n3.Considering the complexity of the proposed algorithm, and its marginal improvement over previous methods on two real datasets in Table 2, especially compared to NCE. The effectiveness of this work is questionable. Additional experiments are suggested, especially on Clothing1M. Besides, some related works[1] should be discussed which are published recently.  \n\n [1] OT-Filter: An Optimal Transport Filter for Learning with Noisy Labels (CVPR 2023)\n\n4.The results of row (g) in Table 3 are not sufficient to show that the performance improvement is brought about by the method in this paper. For example, we need a more detailed ablation study to explain the role of NCE loss and the role of CSOT.\n\n5.The structure of the article is confusing, which weakens the contribution of this article. Secondly, the introduction of some tool concepts is quite abrupt. For example, the proposal of Eq. (3) and (6) gives people a new form of OT formulation and will give specific solutions later. This expectation is affected by the entropy regularization in the following text, whose introduction is abrupt. This results in the final Eq. (16) which does not look significantly different from the original sinkhorn algorithm. It is recommended to introduce sinkhorn from the beginning and to emphasize that there are new constraints based on it.\n\n6.The characters in the figures in the experimental part are too small, which affects reading. At the same time, why are there two figures, and their labels are figure 2? Moreover, Fig. 4 is quoted in the description of the text, but there is no figure 4 in fact.\n\nQUESTIONS:\n1.For more precision, I would like to suggest providing a more detailed description of Equation (6). What does the left constraint in Equation (6) signify? Why should $Q\\mathbb{1}_{C}$ not exceed $\\frac{1}{B}\\mathbb{1}_{B}$? Furthermore, it seems that $\\frac{m}{c}\\mathbb{1}_{C}$ is not a simplex. Could you please explain the rationale behind this relationship and provide additional descriptions or citations to support it? Additionally, the introduction of curriculum into SOT appears to be too direct and lacks smoothness.\n\n2.The identification of selected samples in Equation (7) is confusing. What does \"topK(W,⌊mB⌋)\" mean? Based on my understanding, it implies that a sample pair (x_i, y_i) will be considered clean if it belongs to the top k convincing samples in the current training batch. I would suggest adding a description for the round-down symbol ⌊ and ⌋ to clarify its purpose.\n\nLIMITATIONS:\nThe authors describe its limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CSOT, an approach to address the challenge of noisy labels in machine learning models. CSOT incorporates optimal transport formulation to assign reliable labels during training, considering the structure of the sample distribution. The authors also propose an efficient computational method for solving CSOT. Experimental results demonstrate the superior performance of CSOT compared to existing methods for learning with noisy labels.\n\nSTRENGTHS:\n- The paper is strongly motivated by theoretical analysis, particularly optimal transport analysis.\n- The writing style is clear and easy to follow.\n- CSOT exhibits superior performance when compared to previous algorithms.\n\nWEAKNESSES:\n- The paper lacks a comparison with a baseline algorithm called UNICON [1], which has shown good performance in highly noisy scenarios (e.g., 0.9 noisy ratio). It would be valuable for the authors to include a performance comparison with UNICON.\n- The authors do not analyze the case of instance-wise noisy labels, which is a prevalent type of noisy label model. Including an analysis of this case would be beneficial.\n- The paper does not investigate the sensitivity of hyperparameters, which are required to run the algorithm. It would be valuable for the authors to perform a hyperparameter sensitivity analysis.\n- To enable a comprehensive comparison, the authors should report both the best and last performances of the model, as models trained on noisy labels tend to memorize the noisy labels.\n\n[1] UNICON: Combating Label Noise Through Uniform Selection and Contrastive Learning\n\nMinor)\nThe legend size in Figure 2 is too small to read.\n\nQUESTIONS:\n-\n\nLIMITATIONS:\nThe limitations of this paper are summarized in the \"Question\" and \"Weakness\" sections\n------\n(Raise score from 5 to 6 after rebuttal)", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel optimal transport formulation, called Curriculum and Structure-aware Optimal Transport (CSOT) for learning with noisy labels. CSOT considers both the inter- and intra-distribution structure of the samples to construct a robust denoising and relabeling allocator. It’s worthing noting that Notably, CSOT is a new OT formulation with a nonconvex objective function and curriculum constraints. The authors developed a lightspeed computational method that involves a scaling iteration within a generalized conditional gradient framework to solve CSOT efficiently.\n\nSTRENGTHS:\n1. Novel Approach: The paper proposes a novel optimal transport (OT) formulation called Curriculum and Structure-aware Optimal Transport (CSOT) to address the challenge of learning with noisy labels. I believe this paper introduces a new perspective and potentially brings fresh insights to the field.\n\n2. Consideration of Global and Local Structure: Unlike current approaches that evaluate each sample independently, CSOT concurrently considers the inter- and intra-distribution structure of the samples. This consideration of both global and local structure helps construct a more robust denoising and relabeling allocator, potentially leading to improved performance.\n\n3. Incremental Assignment of Reliable Labels: CSOT incrementally assigns reliable labels to a fraction of the samples with the highest confidence during the training process. This approach ensures that the assigned labels have both global discriminability and local coherence, which could contribute to better generalization and reduced overfitting.\n\n4. This paper provides a very detailed derivation for the lightspeed computational method.\n\nWEAKNESSES:\nResearchers or practitioners interested in using CSOT may need to invest additional effort in adapting or developing specialized solvers.\n\nQUESTIONS:\n1. How does CSOT incorporate the global and local structure of the sample distribution? Could you provide more details on the methodology used to capture this information?\n\n2. Can you explain in more detail the nonconvex objective function of CSOT? How does the nonconvexity affect the optimization process, and how does the proposed method handle this challenge?\n\n3. What is the iteration number required by the generalized conditional gradient? \n\n4. Are there any specific scenarios or types of noisy labels where CSOT may not perform as well?\n\nLIMITATIONS:\nCSOT is described as having a nonconvex objective function. Nonconvex optimization problems can be challenging to solve, and they may have multiple local optima, making it difficult to guarantee finding the global optimum. This could potentially impact the reliability and efficiency of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new noisy label learning approach based on Optimal Transport (OT) and Pseudo-Labeling (PL). Specifically, the authors extent OT-based PL with the consideration of the intrinsic coherence structure of sample distribution. Consequently, this paper proposes a novel optimal transport formulation, namely Curriculum and Structure-aware Optimal Transport (CSOT), which constructs a robust denoising and relabeling allocator that mitigates error accumulation. Experiments on both controlled and real noisy label datasets show the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. The paper proposes a method named Curriculum and Structure-aware Optimal Transport (CSOT) to address the problem of noisy label learning, and the application of OT-based pseudo-labeling in tackling noisy label learning problem has not been thoroughly investigated.\n\n2. The experimental results on different datasets in this paper validate the effectiveness of the proposed method. Additionally, several ablation experiments are conducted to demonstrate the effectiveness of each module in the method.\n\nWEAKNESSES:\n1. In terms of methodological novelty, OT-based PL has been previously applied to other problems, and this paper only applies it to the specific problem namely noisy label learning rather than introducing it for the first time. Additionally, employing curriculum learning to address the issue of pseudo-labeling is a common strategy in the field of weakly supervised learning.\n\n2. The utilization of SOT is a key contribution of this paper. However, the current motivation behind this aspect, as presented in Figure 1, is not sufficiently clear. The authors are encouraged to provide additional descriptions in this section to enhance the clarity and understanding of the motivation.\n\n3. Since the differences between the proposed method and the comparison methods in several cases is too small, it is hard to provide a clear comparison without the present of standard deviations. Additionally, the backbone and other parameter settings of the SOTA methods are not clearly listed, thus, further evidence is needed to establish the fairness of the comparisons.\n\nQUESTIONS:\nPlease refer to Weaknesses\n\nLIMITATIONS:\nN/A.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}