SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where we denote the cost matrix  $\mathbf{C} := -\log \mathbf{P}$  in Probelm (6) for simplicity. Inspired by [8], Problem (12) can be easily re-written as the Kullback-Leibler (KL) projection:  $\min_{\mathbf{Q} \in \Pi^c(\alpha, \beta)} \varepsilon \mathrm{KL}(\mathbf{Q}|e^{-\mathbf{C}/\varepsilon})$ . Besides, the polytope  $\Pi^c(\alpha, \beta)$  can be expressed as an intersection of two convex but not affine sets, i.e.
$$
\mathcal {C} _ {1} \stackrel {\text {d e f}} {=} \left\{\mathbf {Q} \in \mathbb {R} _ {+} ^ {| \boldsymbol {\alpha} | \times | \boldsymbol {\beta} |} \mid \mathbf {Q} \mathbb {1} _ {| \boldsymbol {\beta} |} \leq \boldsymbol {\alpha} \right\} \quad \text {a n d} \quad \mathcal {C} _ {2} \stackrel {\text {d e f}} {=} \left\{\mathbf {Q} \in \mathbb {R} _ {+} ^ {| \boldsymbol {\alpha} | \times | \boldsymbol {\beta} |} \mid \mathbf {Q} ^ {\top} \mathbb {1} _ {| \boldsymbol {\alpha} |} = \boldsymbol {\beta} \right\}. \tag {13}
$$
In light of this, Problem (12) can be solved by performing iterative KL projection between  $\mathcal{C}_1$  and  $\mathcal{C}_2$ , namely Dykstra's algorithm [21] shown in Appendix.
Lemma 1. (Efficient scaling iteration for Curriculum OT) When solving Problem (12) by iterating Dykstra's algorithm, the matrix  $\mathbf{Q}^{(n)}$  at  $n$  iteration is a diagonal scaling of  $\mathbf{K} \coloneqq e^{-\mathbf{C} / \varepsilon}$ , which is the element-wise exponential matrix of  $-\mathbf{C} / \varepsilon$ :
$$
\mathbf {Q} ^ {(n)} = \operatorname {d i a g} \left(\boldsymbol {u} ^ {(n)}\right) \mathbf {K} \operatorname {d i a g} \left(\boldsymbol {v} ^ {(n)}\right), \tag {14}
$$
where the vectors  $\pmb{u}^{(n)}\in \mathbb{R}^{|\pmb{\alpha}|}$ ,  $\pmb{v}^{(n)}\in \mathbb{R}^{|\pmb{\beta}|}$  satisfy  $\pmb{v}^{(0)} = \mathbb{1}_{|\pmb{\beta}|}$  and follow the recursion formula
$$
\boldsymbol {u} ^ {(n)} = \min  \left(\frac {\boldsymbol {\alpha}}{\mathbf {K} \boldsymbol {v} ^ {(n - 1)}}, \mathbb {1} _ {| \boldsymbol {\alpha} |}\right) \quad \text {a n d} \quad \boldsymbol {v} ^ {(n)} = \frac {\boldsymbol {\beta}}{\mathbf {K} ^ {\top} \boldsymbol {u} ^ {(n)}}. \tag {15}
$$
The proof is given in the Appendix. Lemma 1 allows a fast implementation of Dykstra's algorithm by only performing matrix-vector multiplications. This scaling iteration for entropic regularized COT is very similar to the widely-used and efficient Sinkhorn Algorithm [19], as shown in Algorithm 1.
5.2 Solving Curriculum and Structure-Aware Optimal Transport
In the following, we propose to solve CSOT within a Generalized Conditional Gradient (GCG) algorithm [59] framework, which strongly relies on computing Curriculum OT by scaling iterations
Algorithm 1 Efficient scaling iteration for entropic regularized Curriculum OT
1: Input: Cost matrix C, marginal constraints vectors  $\alpha$  and  $\beta$ , entropic regularization weight  $\varepsilon$   
2: Initialize:  $\mathbf{K} \gets e^{-\mathbf{C} / \varepsilon}$ ,  $\pmb{v}^{(0)} \gets \mathbb{1}_{|\beta|}$   
3: Compute:  $\mathbf{K}_{\alpha} \gets \frac{\mathbf{K}}{\operatorname{diag}(\alpha)\mathbb{1}_{|\alpha| \times |\beta|}}$ ,  $\mathbf{K}_{\beta}^{\top} \gets \frac{\mathbf{K}^{\top}}{\operatorname{diag}(\beta)\mathbb{1}_{|\beta| \times |\alpha|}}$  // Saving computation  
4: for  $n = 1, 2, 3, \ldots$  do  
5:  $\pmb{u}^{(n)} \gets \min \left(\frac{\mathbb{1}_{|\alpha|}}{\mathbf{K}_{\alpha}\pmb{v}^{(n-1)}}, \mathbb{1}_{|\alpha|}\right)$   
6:  $\pmb{v}^{(n)} \gets \frac{\mathbb{1}_{|\beta|}}{\mathbf{K}_{\beta}^{\top}\pmb{u}^{(n)}}$   
7: end for  
8: Return:  $\operatorname{diag}(\pmb{u}^{(n)})\mathbf{Kdiag}(\pmb{v}^{(n)})$
in Algorithm 1. The conditional gradient algorithm [27, 36] has been used for some penalized OT problems [24, 17] or nonconvex Gromov-Wasserstein distances [58, 67, 13], which can be used to solve Problem (3) directly.
For simplicity, we denote the local coherent regularized terms as  $\Omega (\cdot)\coloneqq \Omega^{\mathbf{P}}(\cdot) + \Omega^{\mathbf{L}}(\cdot)$ , and give an entropic regularized CSOT formulation as follows:
$$
\min  _ {\mathbf {Q} \in \Pi^ {\mathrm {c}} (\alpha , \beta)} \left\langle \mathbf {C}, \mathbf {Q} \right\rangle + \kappa \Omega (\mathbf {Q}) + \varepsilon \left\langle \mathbf {Q}, \log \mathbf {Q} \right\rangle . \tag {16}
$$
Since the local coherent regularized term  $\Omega^{\mathbf{P}}(\cdot)$  is differentiable, Problem (16) can be solved within the GCG algorithm framework, shown in Algorithm 2. And the linearization procedure in Line 5 can be computed efficiently by the scaling iteration proposed in Sec 5.1.
Algorithm 2 Generalized conditional gradient algorithm for entropic regularized CSOT
1: Input: Cost matrix C, marginal constraints vectors  $\alpha$  and  $\beta$ , entropic regularization weight  $\varepsilon$ , local coherent regularization weight  $\kappa$ , local coherent regularization function  $\Omega: \mathbb{R}^{|\alpha| \times |\beta|} \to \mathbb{R}$ , and its gradient function  $\nabla \Omega: \mathbb{R}^{|\alpha| \times |\beta|} \to \mathbb{R}^{|\alpha| \times |\beta|}$   
2: Initialize:  $\mathbf{Q}^{(0)} \gets \boldsymbol{\alpha} \boldsymbol{\beta}^T$   
3: for  $i = 1, 2, 3, \ldots$  do  
4:  $\mathbf{G}^{(i)} \gets \mathbf{Q}^{(i)} + \kappa \nabla \Omega(\mathbf{Q}^{(i)}) // Gradient computation$   
5:  $\widetilde{\mathbf{Q}}^{(i)} \gets \operatorname*{argmin}_{\mathbf{Q} \in \Pi^c(\alpha, \beta)} \langle \mathbf{Q}, \mathbf{G}^{(i)} \rangle + \varepsilon \langle \mathbf{Q}, \log \mathbf{Q} \rangle$   
// Linearization, solved efficiently by Algorithm 1  
6: Choose  $\eta^{(i)} \in [0, 1]$  so that it satisfies the Armijo rule // Backtracking line-search  
7:  $\mathbf{Q}^{(i+1)} \gets (1 - \eta^{(i)}) \mathbf{Q}^{(i)} + \eta^{(i)} \widetilde{\mathbf{Q}}^{(i)} // Update$   
8: end for  
9: Return:  $\mathbf{Q}^{(i)}$
6.1 Implementation Details
We conduct experiments on three standard LNL benchmark datasets: CIFAR-10 [43], CIFAR-100 [43] and Webvision [49]. We follow most implementation details from the previous work DivideMix [46] and NCE [45]. Here we provide some specific details of our approach. The warm-up epochs are set to 10/30/10 for CIFAR-10/100/Webvision respectively. For CIFAR-10/100, the supervised learning epoch  $T_{sup}$  is set to 250, and the semi-supervised learning epoch  $T_{semi}$  is set to 200. For Webvision,  $T_{sup} = 80$  and  $T_{semi} = 70$ . For all experiments, we set  $\lambda_1 = 1$ ,  $\lambda_2 = 1$ ,  $\varepsilon = 0.1$ ,  $\kappa = 1$ . And we adopt a simple linear ramp for curriculum budget, i.e.  $m = \min(1.0, m_0 + \frac{t - 1}{T_{sup} - 1})$  with an initial budget  $m_0 = 0.3$ . For the GCG algorithm, the number of outer loops is set to 10, and the number for inner scaling iteration is set to 100. The batch size  $B$  for denoising and relabeling is set to 1024. More details will be provided in Appendix.
Table 1: Comparison with state-of-the-art methods in test accuracy (\%) on CIFAR-10 and CIFAR-100. The results are mainly copied from [45, 48]. We present the performance of our CSOT method using the "mean±variance" format, which is obtained from 3 trials with different seeds.
Dataset
Noise type
Method/Noise ratio	CIFAR-10	CIFAR-100
Symmetric	Assymmetric	Symmetric
0.2	0.5	0.8	0.9	0.4	0.2	0.5	0.8	0.9
Cross-Entropy	86.8	79.4	62.9	42.7	85.0	62.0	46.7	19.9	10.1
F-correction [56]	86.8	79.8	63.3	42.9	87.2	61.5	46.6	19.9	10.2
Co-teaching+ [31]	89.5	85.7	67.4	47.9	-	65.6	51.8	27.9	13.7
PENCIL [76]	92.4	89.1	77.5	58.9	88.5	69.4	57.5	31.1	15.3
DivideMix [46]	96.1	94.6	93.2	76.0	93.4	77.3	74.6	60.2	31.5
ELR [50]	95.8	94.8	93.3	78.7	93.0	77.6	73.6	60.8	33.4
NGC [72]	95.9	94.5	91.6	80.5	90.6	79.3	75.9	62.7	29.8
RRL [48]	96.4	95.3	93.3	77.4	92.6	80.3	76.0	61.1	33.1
MOIT [55]	93.1	90.0	79.0	69.6	92.0	73.0	64.6	46.5	36.0
UniCon [41]	96.0	95.6	93.9	90.8	94.1	78.9	77.6	63.9	44.8
NCE [45]	96.2	95.3	93.9	88.4	94.5	81.4	76.3	64.7	41.1
OT Cleaner [73]	91.4	85.4	56.9	-	-	67.4	58.9	31.2	-
OT-Filter [23]	96.0	95.3	94.0	90.5	95.1	76.7	73.8	61.8	42.8
CSOT (Best)	96.6±0.10	96.2±0.11	94.4±0.16	90.7±0.33	95.5±0.06	80.5±0.28	77.9±0.18	67.8±0.23	50.5±0.46
CSOT (Last)	96.4±0.18	96.0±0.11	94.3±0.20	90.5±0.36	95.2±0.12	80.2±0.31	77.7±0.14	67.6±0.36	50.3±0.33
Table 2: Comparison with SOTA methods in top-1 / 5 test accuracy  $(\%)$  on the Webvision and ImageNet ILSVRC12 validation sets.
Method	Webvision	ILSVRC12
top-1	top-5	top-1	top-5
F-correction [56]	61.12	82.68	57.36	82.36
Decoupling [52]	62.54	84.74	58.26	82.26
MentorNet [39]	63.00	81.40	57.80	79.92
Co-teaching [31]	63.58	85.20	61.48	84.70
DivideMix [46]	77.32	91.64	75.20	90.84
ELR [50]	76.26	91.26	68.71	87.84
ELR+ [50]	77.78	91.68	70.29	89.76
NGC [72]	79.20	91.80	74.40	91.00
RRL [48]	77.80	91.30	74.40	90.90
UniCon [41]	77.60	93.44	75.29	93.72
MOIT [55]	77.90	91.90	73.80	91.70
NCE [45]	79.50	93.80	76.30	94.10
CSOT	79.67	91.95	76.64	91.67
Table 3: Time cost (s) for solving CSOT optimization problem of different input sizes. VDA indicates vanilla Dykstras algorithm-based CSOT solver, while ESI indicates the efficient scaling iteration-based solver.
(|α|, |β|)	VDA-based	ESI-based (Ours)
(1024,10)	0.83	0.82↓
(1024,50)	1.00	0.80↓
(1024,100)	0.87	0.80↓
(50,50)	0.82	0.79↓
(100,100)	0.88	0.80↓
(500,500)	0.88	0.87↓
(1000,1000)	0.94	0.81↓
(2000,2000)	2.11	0.98↓
(3000,3000)	3.74	0.99↓
6.2 Comparison with the State-of-the-Arts
Synthetic noisy datasets. Our method is validated on two synthetic noisy datasets, i.e. CIFAR-10 [43] and CIFAR-100 [43]. Following [46, 45], we conduct experiments with two types of label noise: symmetric and asymmetric. Symmetric noise is injected by randomly selecting a percentage of samples and replacing their labels with random labels. Asymmetric noise is designed to mimic the pattern of real-world label errors, i.e. labels are only changed to similar classes (e.g. cat  $\leftrightarrow$  dog). As shown in Tab. 1, our CSOT has surpassed all the state-of-the-art works across most of the noise ratios. In particular, our CSOT outperforms the previous state-of-the-art method NCE [45] by  $2.3\%$ ,  $3.1\%$  and  $9.4\%$  under a high noise rate of CIFAR-10 sym-0.8, CIFAR-100 sym-0.8/0.9, respectively.
Real-world noisy datasets. Additionally, we conduct experiments on a large-scale dataset with real-world noisy labels, i.e. WebVision [49]. WebVision contains 2.4 million images crawled from the web using the 1,000 concepts in ImageNet ILSVRC12 [20]. Following previous works [46, 45], we conduct experiments only using the first 50 classes of the Google image subset for a total of  $\sim 61,000$  images. As shown in Tab. 2, our CSOT surpasses other methods in top-1 accuracy on both Webvision and ILSVRC12 validation sets, demonstrating its superior performance in dealing with real-world noisy datasets. Even though NCE achieves better top-5 accuracy, it suffers from high time costs (using a single NVIDIA A100 GPU) due to the co-training scheme, as shown in Tab. S5.
Table 4: Ablation studies under multiple label noise ratios on CIFAR-10 and CIFAR-100. "repl." is an abbreviation for "replaced", and  $\mathcal{L}^{ce}$  represents a cross-entropy loss. GMM refers to the selection of clean labels based on small-loss criterion [46]. CT (confidence thresholding [62]) is a relabeling scheme where we set the CT value to 0.95.
Dataset	CIFAR-10	CIFAR-100	Avg
Noise type	Sym.	Asym.	Sym.
Method/Noise ratio	0.5	0.8	0.9	0.4	0.5	0.8	0.9
Denoise Relabeling Technique	(a) Classical OT	95.45	91.95	82.35	95.04	75.96	62.46	43.28	78.07
(b) Structure-aware OT	95.86	91.87	83.29	95.06	76.20	63.73	44.57	78.65
(c) CSOT w/o ΩP and ΩL	95.53	93.84	89.50	95.14	75.96	66.50	47.55	80.57
(d) CSOT w/o ΩP	95.77	94.08	89.97	95.35	76.09	66.79	48.13	80.88
(e) CSOT w/o ΩL	95.55	93.97	90.41	95.15	76.17	67.28	48.01	80.93
Learning Technique	(f) GMM + Lsup	92.48	80.37	31.76	90.80	69.52	48.49	20.86	62.04
(g) CSOT repl. Lsup with Lce	93.47	81.93	53.45	91.43	72.66	50.62	21.77	66.48
(h) CSOT w/o Lsemi	95.34	93.04	88.9	94.11	75.16	61.13	36.94	77.80
(i) CSOT repl. correction with CT (0.95)	95.46	90.73	89.09	95.21	75.85	64.28	48.76	79.91
(j) CSOT w/o Lsimsiam	95.92	94.17	89.31	95.16	76.38	66.17	45.56	80.38
CSOT	96.20	94.39	90.65	95.50	77.94	67.78	50.50	81.85
6.3 Ablation Studies and Analysis
Effectiveness of CSOT-based denoising and relabeling. To verify the effectiveness of each component in our CSOT, we conduct comprehensive ablation experiments, shown in Tab. 4. Compared to classical OT, Structure-aware OT, and Curriculum OT, our proposed CSOT has achieved superior performance. Specifically, our proposed local coherent regularized terms  $\Omega^{\mathbf{P}}$  and  $\Omega^{\mathbf{Q}}$  indeed contribute to CSOT, as demonstrated in Tab 4 (c)(d)(e). Furthermore, our proposed curriculum constraints yield an improvement of approximately  $2\%$  for both classical OT and structure-aware OT, as shown in Tab 4 (a)(b)(c). Particularly, under high noise ratios, the improvement can reach up to  $4\%$ , which demonstrates the effectiveness of the curriculum relabeling scheme.
Effectiveness of clean labels identification via CSOT. As shown in Tab. 4 (f), replacing our CSOT-based denoising and relabeling with GMM [46] for clean label identification significantly degrades the model performance. This phenomenon can be explained by the clean accuracy during training (Fig. 2a) and clean recall rate (Fig. 2c), in which our CSOT consistently outperforms other methods in accurately retrieving clean labels, leading to significant performance improvements. These experiments fully show that our CSOT can maintain both high quantity and high quality of clean labels during training.
Effectiveness of corrupted labels correction via CSOT. As shown in Tab. 4 (h), only training with identified clean labels leads to inferior model performance. Furthermore, replacing our CSOT-based denoising and relabeling with confidence thresholding (CT) [62] for corrupted label correction also degrades the model performance, as shown in Tab. 4 (i). The CT methods assign pseudo labels to samples based on model prediction, which is unreliable in the early training stage, especially under high noise rates. Our CSOT-based denoising and relabeling fully consider the inter- and intra-distribution structure of samples, yielding more robust labels. Particularly, our CSOT outperforms NCE and DivideMix significantly in label correction, as demonstrated by the superior corrected accuracy in Fig. 2b and the improved clarity of the confusion matrix in Fig. S7.
Effectiveness of curriculum training scheme. According to the progressive clean and corrupted accuracy during the training process shown in Fig. 2a and Fig. 2b, our curriculum identification scheme ensures high accuracy in the early training stage, avoiding overfitting to wrong corrected labels. Note that since our model is trained using only a fraction of clean samples, it is crucial to employ a powerful supervised learning loss to facilitate better learning. Otherwise, the performance will be poor without the utilization of a powerful supervised training loss, as evidenced in Tab. 4 (g). In addition, the incorporation of self-supervised loss enhances noise-robust representation, particularly in high noise rate scenarios, as demonstrated in our experiments in Tab. 4 (j).
Time cost discussion for solving CSOT To verify the efficiency of our proposed lightspeed scaling iteration, we conduct some experiments for solving CSOT optimization problem of different input sizes on a single GPU NVIDIA A100. As demonstrated in Tab. 3, our proposed lightspeed
(a) Clean accuracy
(b) Corrected accuracy
Figure 2: Performance comparison for clean label identification and corrupted label correction.
(c) Clean recall rate
computational method that involves an efficient scaling iteration (Algorithm 1) achieves lower time cost compared to vanilla Dykstras algorithm (Algorithm S6). Specifically, compared to the vanilla Dykstra-based approach, our efficient scaling iteration version can achieve a speedup of up to 3.7 times, thanks to efficient matrix-vector multiplication instead of matrix-matrix multiplication. Moreover, even for very large input sizes, the computational time cost does not increase significantly.
In this paper, we proposed Curriculum and Structure-aware Optimal Transport (CSOT), a novel solution to construct robust denoising and relabeling allocator that simultaneously considers the inter- and intra-distribution structure of samples. Unlike current approaches, which rely solely on the model's predictions, CSOT considers the global and local structure of the sample distribution to construct a robust denoising and relabeling allocator. During the training process, the allocator assigns reliable labels to a fraction of the samples with high confidence, ensuring both global discriminability and local coherence. To efficiently solve CSOT, we developed a lightspeed computational method that involves a scaling iteration within a generalized conditional gradient framework. Extensive experiments on three benchmark datasets validate the efficacy of our proposed method. While class-imbalance cases are not considered in this paper within the context of LNL, we believe that our approach can be further extended for this purpose.
This work was supported by NSFC (No.62303319), Shanghai Sailing Program (21YF1429400, 22YF1428800), Shanghai Local College Capacity Building Program (23010503100), Shanghai Frontiers Science Center of Human-centered Artificial Intelligence (Shanghai), MoE Key Laboratory of Intelligent Perception and Human-Machine Collaboration (ShanghaiTech University), and Shanghai Engineering Research Center of Intelligent Vision and Imaging.