Enhancing Robustness in Learning with Noisy Labels: An Asymmetric Co-Training Approach

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-18 08:53:23

Abstract: Label noise, an inevitable issue in various real-world datasets, tends to impair the performance of deep neural networks.
A large body of literature focuses on symmetric co-training, aiming to enhance model robustness by exploiting interactions between models with distinct capabilities. However, the symmetric training processes employed in existing methods often culminate in model consensus, diminishing their efficacy in handling noisy labels. To this end, we propose an Asymmetric Co-Training (ACT) method to mitigate the detrimental effects of label noise. Specifically, we introduce an asymmetric training framework in which one model (i.e., RTM) is robustly trained with a selected subset of clean samples while the other (i.e., NTM) is conventionally trained using the entire training set. We propose two novel criteria based on agreement and discrepancy between models, establishing asymmetric sample selection and mining.
Moreover, a metric, derived from the divergence between models, is devised to quantify label memorization, guiding our method in determining the optimal stopping point for sample mining. Finally, we propose to dynamically re-weight identified clean samples according to their reliability inferred from historical information. We additionally employ consistency regularization to achieve further performance improvement. Extensive experimental results on synthetic and real-world datasets demonstrate the effectiveness and superiority of our method.

Main paper: 
 related work 2.1 learning with noisy labels: 
 Researchers have explored various robust training strategies for learning with noisy labels (LNL) [3, 6, 7, 15, 19, 24, 25, 29, 54, 65] . Existing LNL methods can be categorized into three main directions: sample selection [13, 21, 23, 51] , label correction [1, 6, 11, 21, 52, 58] , and sample re-weighting [9, 34, 43, 44, 46, 63] .Sample Selection: To cope with noisy labels, one intuitive idea is to select clean samples and discard noisy ones from training [23, 53, 56] . Previous sample selection methods primarily regard samples with small losses as clean ones [13, 28, 45, 61] . For instance, DivideMix [21] extracts the clean subset by fitting the loss distribution with the Gaussian Mixture Model. Some recent methods propose new selection criteria for finding clean samples [20, 30] . For example, NCE [20] resorts to neighbor data to identify clean and noisy samples. BARE [30] proposes a data-dependent, adaptive sample selection strategy that relies on batch statistics of a given mini-batch. However, these methods usually demand pre-defined drop rates or thresholds to facilitate efficient selection. Label Correction: Another straightforward idea for addressing noisy labels is to correct corrupted labels before feeding them into networks [1, 6, 11, 12, 31, 52] . Label correction methods typically attempt to rectify sample labels using the noise transition matrix [11] or model predictions [21] . For example, Goldberger et al. [11] proposes to use an additional layer to estimate the noise transition matrix. Jo-SRC [56] uses the temporally averaged model (i.e., meanteacher model) to generate reliable pseudo-label distributions for providing supervision. However, the noise transition matrix is difficult to estimate accurately, while prediction-based label correction tends to suffer from error accumulation. Sample Re-weighting: Recently, some researchers have focused on re-weighting training samples to cope with noisy labels [9, 34, 36, 42, 44] . For example, DIW [9] proposes a dynamic importance weighting strategy as an end-to-end solution to alleviate the bias of static importance weighting. RPM [44] proposes a Bayesian method that infers the example weights as latent variables. L2RW [34] proposes to assign different sample weights based on metalearning. However, existing sample re-weighting methods also tend to require dataset-dependent prior knowledge (e.g., a small subset of clean samples), posing a limit to their practicability.Symmetric co-training is one of the most frequently-employed strategies in sample selection methods [13, 21, 28, 38, 40, 45, 56, 61] . The idea of SCT stems from the Co-training approach [4] , which aims to obtain information gains by simultaneously training two models and enabling them to mutually guide the learning process. In SCT methods, the two models have identical architectures but are initialized differently to acquire discrepant learning capabilities.For instance, Co-teaching [13] trains two networks simultaneously and selects small-loss data to teach the peer network during training. JoCoR [45] maintains two networks, training them with a joint loss to make their predictions converge. Co-learning [40] proposes to train a shared feature encoder with two distinctive prediction heads, maximizing their agreement in the latent space. Co-LDL [38] simultaneously trains two models and lets them communicate useful knowledge by selecting low-loss and high-loss samples for each other. However, our study posits that the additional information gain introduced by the SCT strategy is constrained, as the disparities between the two models primarily arise from random initialization. The dual models will eventually converge, leading to a diminution of their effectiveness in addressing noisy labels. 
problem statement: 
 Considering a classification problem with 𝐶 classes, let us suppose that X ⊂ R 𝑑 is the input space and Y = {0, 1} 𝐶 is the given label space (in a one-hot manner). We denote 𝐷 = {(𝑥, 𝑦)|𝑥 ∈ X, 𝑦 ∈ Y} as the training set, which is obtained from the joint distribution over X × Y. For noisy label learning, the given label 𝑦 ∈ Y is potentially "incorrect" and we use 𝑦 * to represent the ground-truth label of the sample 𝑥. In conventional supervised learning, the DNN learns a mapping function F : X → Y on the training set 𝐷 and optimizes the network parameters 𝜃 using the following cross-entropy loss:EQUATION)The goal is to obtain optimal parameters 𝜃 * by minimizing the empirical risk R L (F ) subjected to network parameters as follows:EQUATIONGiven the remarkable fitting capability of DNNs [62] , optimization of network parameters using noisy labels within the conventional supervised learning framework can potentially steer the model toward an undesirable direction. Therefore, it is imperative to establish a solution capable of effectively addressing noisy labels.SCT has been demonstrated effective in learning with noisy labels, particularly in sample selection-based methods [13, 21, 28, 40, 45, 56, 61] . Resorting to the simultaneously trained dual networks, SCT effectively harnesses their diverse learning capabilities to promote model robustness in a mutual-reinforced manner. However, the two models in SCT are destined to converge due to the identical network architecture and the homogeneous training process, eventually vanishing the information gains obtained from symmetric training.To this end, we propose an Asymmetric Co-Training (ACT) method, aiming to continuously enhance model robustness against noisy labels through asymmetric learning. In contrast to SCT, where both models adhere to the same training process, our ACT simultaneously trains two networks (i.e., RTM and NTM) with identical architectures but employs distinct training strategies. RTM (i.e., 𝜃 𝑅𝑇 𝑀 ) adopts a robust training strategy during network optimization, while NTM (i.e., 𝜃 𝑁𝑇 𝑀 ) is trained with the entire training set following the conventional supervised learning process. As such, an asymmetric co-training framework is accordingly established. Specifically, to facilitate the robustness against noisy labels, we perform loss back-propagation only on a selected "clean" subset 𝐷 𝑐 ⊆ 𝐷 when training RTM. Its loss function is as follows:EQUATIONFor the NTM, we follow the conventional supervised learning pro- function is as follows:Dŷ R = y, ŷN = y Eŷ R y, ŷN = y Fŷ R = y, ŷN y Gŷ R y, ŷN y 6HOHFWLRQ3HULFLVLRQ 3UHGLFWLRQ$FFXUDF\EQUATIONAccordingly, given an input sample 𝑥, we can derive the prediction results produced by the two models as:EQUATION𝑝 (𝑥, 𝜃 𝑅𝑇 𝑀 ) 𝑐 and 𝑝 (𝑥, 𝜃 𝑁𝑇 𝑀 ) 𝑐 represent the prediction probabilities of the sample 𝑥 on the category 𝑐 by models 𝜃 𝑅𝑇 𝑀 and 𝜃 𝑁𝑇 𝑀 .Obviously, the NTM is predetermined to overfit noisy samples and yield degenerated performance. However, our design enables the NTM to complement the RTM with knowledge learned from a different perspective. Specifically, based on our asymmetric training design, the RTM consistently engages in robust learning from clean samples, whereas the NTM progressively fits all samples (including noisy ones) as a result of label memorization. RTM and NTM tend to exhibit agreement when learning clean samples (i.e., robust learning) but disagreement when learning noisy samples (i.e., label memorization). Consequently, we argue that our asymmetric training can provide more unique insights for selecting clean samples compared to existing SCT methods.Existing SCT methods have investigated both agreement-based [13, 45] and disagreement-based [28, 61] sample selection strategies for addressing noisy labels. However, their reliabilities are prone to be compromised due to the converging behavior of SCT models. Especially in the later training stage, SCT models tend to produce consentaneous predictions even when confronted with noisy data. This diminishes their precision in selecting clean samples, thereby leading to degraded model performance. Inspired by existing agreement-based and disagreement-based sample selection methods, Our ACT revisits the relationship between predictions and given labels. By employing our asymmetric training design, we obtain insights that support us in devising better sample selection criteria, aiding in the selection and mining of more valuable and reliable clean samples for RTM.Specifically, we find the relationships between predictions of models (i.e., ŷ𝑅 and ŷ𝑁 ) and given labels 𝑦 can be categorized into four situations: (1) ŷ𝑅 = 𝑦 and ŷ𝑁 = 𝑦, (2) ŷ𝑅 ≠ 𝑦 but ŷ𝑁 = 𝑦, (3) ŷ𝑅 = 𝑦 but ŷ𝑁 ≠ 𝑦, and (4) ŷ𝑅 ≠ 𝑦 and ŷ𝑁 ≠ 𝑦. As depicted in Fig. 3 , we compare the selection precision and the prediction accuracy of corresponding samples w.r.t. their ground truth by conducting experiments on a synthetically noisy dataset. Fig. 3 (a) demonstrates the high selection precision (approaching 0.998) in situation (1) . As the samples selected by situation (1) exhibit agreement between given labels and model predictions, it ensures the high accuracy of RTM predictions. Inspired by the results in Fig. 3 (a), we introduce a new criterion to select clean samples for RTM as follows: Criterion 1. A sample 𝑥 is deemed clean if its predicted results of RTM and NTM are consistent and aligned with its given label 𝑦 (i.e., ŷ𝑅 = ŷ𝑁 = 𝑦).Therefore, the clean subset 𝐷 𝑐 in our ACT that we select to participate in the training of 𝜃 𝑅𝑇 𝑀 and the corresponding noisy subset is defined as follows:𝐷 𝑐 = {(𝑥, 𝑦)|(𝑥, 𝑦) ∈ 𝐷, ŷ𝑅 = ŷ𝑁 = 𝑦}, 𝐷 𝑛 = 𝐷 -𝐷 𝑐 . (6)Fig. 3 (b), (c), and (d ) depict the selection precision and prediction accuracy for the latter three situations, where the predictions of models and given labels exhibit disagreement. In Fig. 3 (b) , we observe the selection begins with high precision but exhibits a notable decreasing trend. Meanwhile, the prediction accuracy of 𝜃 𝑅𝑇 𝑀 is consistently low. Essentially, this case has the potential to mine additional clean samples that 𝜃 𝑅𝑇 𝑀 has not yet learned (i.e., ŷ𝑅 ≠ 𝑦). In Fig. 3 (c ), both the selection precision and the prediction accuracy are consistently high. This indicates that 𝜃 𝑅𝑇 𝑀 adeptly fits this subset of samples, meaning that little additional information can be unearthed from this subset. Results in Fig. 3 (d) demonstrate that this portion of data is not conducive to the robustness of 𝜃 𝑅𝑇 𝑀 .Indeed, the samples identified in scenario (2) hold greater importance for mining additional valuable clean samples to enhance the robust training of the RTM before the NTM starts to suffer from label memorization. Since 𝜃 𝑁𝑇 𝑀 will memorize labels to fit noisy samples in the later training stage, the training accuracy of the two models gradually deviates. Accordingly, we design the following self-adaptive metric to measure the extent of label memorization for 𝜃 𝑁𝑇 𝑀 :T = 𝐴𝑐𝑐 (𝜃 𝑁𝑇 𝑀 ) -𝐴𝑐𝑐 (𝜃 𝑅𝑇 𝑀 ) 𝐴𝑐𝑐 (𝜃 𝑅𝑇 𝑀 ) , 𝐴𝑐𝑐 (𝜃 ) = 1 |𝐷 | ∑︁ (𝑥,𝑦) ∈𝐷 𝑦 = ŷ𝜃 . (7)Inspired by the findings from Fig. 3 (b) and T , we additionally introduce a novel criterion for mining more valuable clean samples: Criterion 2. A sample 𝑥 will be mined as a clean sample if its given label 𝑦 does not match the prediction of RTM, yet aligns with that of NTM (i.e., ŷ𝑅 ≠ 𝑦, ŷ𝑁 = 𝑦) before NTM starts to suffer from label memorization (i.e., T ≤ 𝜏).Once the condition T > 𝜏 is triggered, samples selected by ŷ𝑅 ≠ 𝑦, ŷ𝑁 = 𝑦 are no longer reliable and thus should be neglected from the training of RTM. Formally, the subset of selected and mined clean samples at the 𝐾-th epoch is defined as:EQUATIONSome previous works have revealed that determining the cleanness of samples solely based on the current model predictions could bring potential risks in data reliability. The challenge arises from the inevitable fluctuations in model training, making it difficult to prevent a few noisy samples from being leaked into 𝐷 ′ 𝑐 , especially in scenarios with high noise rates.To guarantee the efficacy of our ACT method, we further propose a dynamic sample re-weighting approach to foster the reliability of the selected and mined "clean" samples in 𝐷 ′ 𝑐 . Specifically, we introduce a memory bank (M) to store the selection results of all samples throughout the training process as follows:EQUATIONM 𝐾 (𝑥) denotes the number of epochs that 𝑥 falls into 𝐷 ′ 𝑐 at the 𝐾-th epoch. (M 0 (𝑥) = 0, 0 ≤ M 𝐾 (𝑥) ≤ 𝐾.) Subsequently, we leverage the stored value of M for each clean sample as a weight coefficient when training 𝜃 𝑅𝑇 𝑀 . The loss L 𝜃 𝑅𝑇 𝑀 used for the RTM can be re-write as:EQUATIONin which 𝐾 denotes the 𝐾-th epoch in the training process. Notably, existing sample selection methods often require datasetdependent prior knowledge [13, 45] (e.g., a pre-defined drop rate or threshold). This nature makes it challenging to swiftly adapt them to different real-world scenarios. In contrast, our ACT employs a data-driven, self-adaptive sample selection strategy, rendering it free from dataset-dependent priors. Thus, it is more suitable for real-world applications. Moreover, by incorporating the reliability of sample selection and mining into loss re-weighting, the risk of overfitting to noisy labels is further mitigated, resulting in improved model performance.The training set 𝐷, the robust and non-robust networks 𝜃 𝑅𝑇 𝑀 and 𝜃 𝑁𝑇 𝑀 , warm-up epochs 𝐸 𝑤 , total epochs 𝐸 𝑡𝑜𝑡𝑎𝑙 , batch size 𝑏𝑠. Select "clean" samples using Eq. ( 6); 13:Mine more "clean" samples using Eq. ( 8); 14:Re-weight samples in 𝐷 ′ 𝑐 using Eq. ( 9); 15:Calculate L 𝜃 𝑅𝑇 𝑀 and L 𝜃 𝑁𝑇 𝑀 using Eqs. ( 11) and ( 4); end if 19: end for Output: The updated robust network 𝜃 𝑅𝑇 𝑀 .In summary, we introduce a novel asymmetric co-training approach to alleviate the harmful effects of noisy labels. We simultaneously train two models with identical architectures following different training processes. The RTM is trained with a selected clean subset, while the NTM is trained using the entire noisy training set. We introduce two novel criteria to select and mine clean samples more precisely. A metric is developed to evaluate the degree of label memorization for the NTM, enabling our method to perform mining only before the NTM starts to memorize noisy labels. Moreover, we propose a dynamic sample re-weighting strategy, incorporating the reliability of sample selection and mining to further boost the model performance. The overall learning procedure of our ACT is illustrated in Fig. 2 and Algorithm 1. In practice, we follow [39, 56] and further employ a consistency regularization loss for optimizing 𝜃 𝑅𝑇 𝑀 . Our final objective loss function for RTM is as follows:EQUATION)where 𝜆 is the weighting factor. L 𝑅𝐸𝐺 denotes the consistency regularization (CR) loss, which encourages prediction consistency between weakly-augmented (𝐴 𝑊 ) and strongly-augmented (𝐴 𝑆 ) views of the input samples:EQUATIONin which EQUATION) 
experiments: 
 In this section, we first evaluate the effectiveness of ACT on various synthetic datasets. Then, we perform experiments on real-world benchmark datasets. Finally, we conduct ablation studies to investigate each ingredient in ACT. Further studies, such as the analysis of hyper-parameters, are provided in our supplementary material.Synthetic Datasets: Following [56] , we evaluate our ACT approach on two synthetic datasets (i.e., CIFAR100N and CIFAR80N). CIFAR100N and CIFAR80N originate from CIFAR100 [17] . They are created to simulate closed-set and open-set noisy scenarios, respectively. Adhering to [56] , we primarily study two types of synthetic label noise: symmetric (Sym.) and asymmetric (Asym.).Real-world Datasets: Web-Aircraft, Web-Bird, and Web-Car [37] are three real-world noisy datasets whose training images are crawled from web image search engines. In comparison to synthetic datasets, they present more significant challenges due to their unpredictable noise patterns. Moreover, it has been revealed that they contain both closed-set and open-set noise. Food-101N [18] is another benchmark dataset containing 101 food categories. It comprises around 310k noisy training images. The noise rate and structure are both unknown. Implementation Details: We follow [56] to conduct experiments on synthetic datasets using a seven-layer CNN network as the backbone of our RTM and NTM. Accordingly, models are trained using SGD with a momentum of 0.9 for 150 epochs (including 50 warm-up epochs). To further promote the asymmetricity between the two models, we set the learning rates for the RTM and NTM as 0.01 and 0.08, respectively. The batch size is 128, and the learning rates decay in a cosine annealing manner. When experimenting on real-world datasets, we leverage ResNet50 [14] pre-trained on ImageNet-1K as our backbone. The batch size, the initial learning rate, and the weight decay are 16, 0.005, and 0.0005, respectively. Evaluation Metrics: We adopt test accuracy as the primary metric to assess our model performance. Moreover, to enable a more comprehensive analysis, we additionally evaluate the results of sample selection by using the precision, recall, and F1 score metrics. Our reported performances are averaged results of five repeated runs.Baselines: For synthetic datasets, we compare our ACT with following SOTA methods: Decoupling [28] , Co-teaching [13] , Co-teaching+ [61] , JoCoR [45] , DivideMix [21] , Jo-SRC [56] , Co-LDL [38] , UNICON [16] , SOP [26] , AGCE [66] , DISC [23] , ANL [57] and NPN [35] . For real-world datasets, we additionally compare ACT with other competing methods (e.g., PENCIL [58] , AFM [32] , PLC [64] and DivideMix+SNSCL [47] ). Moreover, we perform conventional training using the entire noisy dataset as a baseline (denoted as Standard). Results of SOTA methods in Tables 1, 2 and 3 are mainly obtained from [56] , [38] and [40] .Table 1 presents the comparison results on the synthetic datasets (i.e., CIFAR100N and CIFAR80N) under various noise types (i.e., symmetric and asymmetric) and noise rates (i.e., 20%, 40% and 80%).Observing Table 1 , we find it is evident that our ACT consistently outperforms all competing methods in various noisy conditions on these synthetic noisy datasets. Especially on CIFAR100N, the performances of our ACT excel existing approaches by notable margins (i.e., 2.75%↑ on Sym-20%, 6.51%↑ on Sym-80%, and 6.37%↑ on Asym-40%), verifying the effectiveness of our method in coping with various closed-set noisy labels. Compared to CIFAR100N, CI-FAR80N is undoubtedly more challenging since it is generated to mimic real-world cases where closed-set and open-set noisy labels simultaneously exist. Our ACT remains the top performer when compared with competing approaches on CIFAR80N. Although our method only achieves 0.35% performance improvement compared to the second-best counterpart (i.e., DISC [23] ) on CIFAR80N (Sym-80%), our ACT obtains remarkable performance gains in the other two cases (i.e., 3.31%↑ on Sym-20% and 5.90%↑ on Asym-40%). This substantiates the efficacy of our proposed ACT method in adeptly tackling diverse challenging noisy labels.To further demonstrate the efficacy of our ACT, we additionally investigate the performance of our asymmetric sample selection and mining by performing a comparison of sample identification results with existing SCT methods (i.e., Decoupling, Co-teaching, Co-teaching+, and JoCoR), using the precision, recall, and F1 score metrics. Fig. 4 shows the comparison results on CIFAR100N with Sym-50% label noise. From Fig. 4 (a), it is evident that the selection precision of our ACT is significantly higher than that of other SCT methods. This observation suggests that the clean samples Table 2 shows the comparison result between our ACT and existing SOTA methods on three real-world datasets (i.e., Web-Aircraft, Web-Bird, and Web-Car). These datasets contain at least 25% of unknown noisy labels and do not provide any label verification information, rendering them both practical and challenging. Table 2 shows that our ACT consistently outperforms these competing methods. Specifically, ACT achieves 86.56%, 81.43%, and 88.75% accuracy on Web-Aircraft, Web-Bird, and Web-Car, respectively, surpassing the second-best performer DISC [23] by 1.29%, 0.35%, and 0.44%. The average test accuracy outperforms DISC by 0.69%. In particular, compared with classic SCT methods (i.e., Decopuling, Co-teaching, Co-teaching+, JoCoR, and Co-LDL), ACT achieves an evidently significant performance improvement. The results, as depicted in Table 2 , provide evidence for the robustness and generalization ability of our ACT method in handling real-world noisy labels. Table 3 presents the performance comparison with SOTA methods on the Food101N dataset. As shown in Table 3 , ACT achieves the best score and outperforms the state-of-the-art SNSCL [47] by 0.41%, validating the effectiveness of our approach in dealing with large-scale, real-world noisy cases.This section, as illustrated in Table 4 , investigates the effectiveness and impact of each ingredient (ASS, ASM, DSRW, and CR) in our method through ablation studies. Standard represents the conventional forward training using the cross-entropy loss. ASS and ASM denote the asymmetric sample selection and mining in our ACT. DSRW indicates the dynamic sample re-weighting process. CR means consistency regularization. Effects of Asymmetric Sample Selection and Mining: Existing SCT methods often face challenges of model convergence, limiting the knowledge acquired from model interactions. In our framework, we introduce two criteria (i.e., Criteria 1 and 2) and formulate the asymmetric sample selection (ASS) and mining (ASM) strategy based on the relationship between model predictions and given labels. From the second row (#2) of Table 4 , we can observe a striking and consistent performance improvement when employing our proposed ASS module. This confirms the capability of ASS in selecting clean samples from the consensus between RTM and NTM. Moreover, as depicted in the third row (#3) of Table 4 , we can find that our proposed ASM also achieves remarkable performance gains. This validates the effectiveness of ASM in mining more valuable clean samples from the discrepancy between the two models. Effects of Dynamic Sample Re-weighting: Due to the lack of ground truth for noisy labels, the identified "clean" samples are never necessarily reliable. Therefore, in our ACT method, we propose a dynamic sample re-weighting (DSRW) module that incorporates the reliability of selected clean samples in the process of loss weighting. DSRW introduces a surrogate metric to measure the reliability of selected samples based on training history, determining their weights in loss back-propagation. This further boosts the robustness against label noise. Table 4 demonstrates that DSRW brings consistent benefit to the model performance. Effects of Consistency Regularization: Our proposed ACT employs consistency regularization (CR) to pursue additional performance gains. Consistency regularization enables us to unearth more knowledge from samples, including those discarded "unclean" and "unmined" ones. Consequently, as shown in Table 4 , our model performance is further promoted. 
conclusion: 
 In this paper, we proposed an asymmetric co-training (ACT) method to address noisy labels. ACT trained two models (i.e., RTM and NTM) simultaneously in an asymmetric manner, equipping them with distinctive capabilities. We accordingly introduced an asymmetric sample selection and mining strategy to reliably identify and mine valuable clean samples. We established a metric based on the divergence between RTM and NTM to quantify label memorization, thereby guiding our ACT on the optimal juncture to cease sample mining. Moreover, a dynamic sample re-weighting scheme was proposed to incorporate the reliability of selected samples in the loss re-weighting process. Comprehensive experiments and ablation studies on various noisy datasets substantiated the effectiveness and superiority of our approach. 
: 
 large volumes of accurate human annotations are costly and timeconsuming to acquire, especially for tasks that necessitate expert annotating knowledge (e.g., medical images [49] ). To obtain largescale annotated data under a limited budget, recent researchers have started to pay attention to using crowd-sourcing platforms [48] or web image search engines [10] for dataset construction. Unfortunately, these methods inevitably introduce low-quality samples with noisy labels, which can cause DNNs to overfit misleading information and degrade their performance. Consequently, developing robust methods aimed at alleviating the detrimental impact of noisy labels is of significant importance.Prior literature has illustrated the Memorization Effect [2] of DNNs, which suggests that models tend to first learn clean samples and then progressively memorize noisy ones. Accordingly, researchers explore a diversity of robust learning strategies, such as sample selection [6, 7, 22, 50, 53, 55] , label correction [41, 58, 59] and sample re-weighting [9, 21, 44] , to mitigate the harmful effects of noisy labels. Notably, among existing solutions, the symmetric co-training (SCT) is one of the most popular training strategies within the realm of sample-selection methods [13, 28, 38, 40, 45, 61] .SCT methods usually entail the simultaneous training of two networks with identical architectures but distinct weight initialization. The twin networks adopt the same training strategy, capitalizing on their distinct learning capabilities to provide mutual guidance throughout the learning process, as shown in Fig. 1 (a) . For example, Decoupling [28] trains two networks simultaneously and updates them using instances with different predictions. Co-teaching [13] maintains two networks simultaneously and enables them to select low-loss samples for each other. Co-teaching+ [61] follows a similar scheme as Co-teaching but proposes to select small-loss data from disagreement one. JoCoR [45] employs a joint loss to select lowloss data, encouraging agreement between networks. The efficacy of SCT methods primarily relies on the assumption that the two networks can extract divergent knowledge from the training data, thereby augmenting robustness through complementary information. However, we argue that the information gains attributed to SCT are substantially constrained since the capability discrepancies between the twin networks mainly arise from distinct initializations. Furthermore, it is problematic that the learning capabilities of the twin networks tend to converge in the later stage of training, leading to a decline in effectiveness for addressing noisy labels [40] .To alleviate aforementioned issues, we propose a novel approach, termed ACT (Asymmetric Co-Training), to combat noisy labels, as shown in Fig. 1 (b ). In our ACT approach, two models with identical architectures are simultaneously trained utilizing distinct training strategies. The first model, designated as the Robustly Trained Model (RTM), is trained with a selected clean subset. RTM by capitalizing on the diverse capabilities of the two models. In our framework, we introduce two novel criteria to devise an asymmetric sample selection and mining strategy that hinges on the relationship between model predictions and given labels, focusing on both consensus and disagreement. Moreover, we propose a dynamic sample re-weighting approach to leverage the historical states throughout the training process, enhancing the reliability of our clean sample selection and mining. Employing two asymmetrically trained models, our ACT establishes a positive feedback loop that continuously promotes the model's robustness against noisy labels. Notably, this enhancement is achieved without the requirement for any dataset-dependent prior knowledge (e.g., a pre-defined noise rate and a small subset of clean samples). Comprehensive experimental results have been provided to verify the effectiveness and superiority of our approach. Our main contributions are summarized as follows:(1) We propose a novel asymmetric co-training (ACT) approach to mitigate the negative impact induced by noisy labels. It trains two networks asymmetrically to improve the reliability of learned knowledge. Through this asymmetric training framework, our RTM and NTM can provide more distinctive insights for clean sample selection compared to existing SCT methods.(2) We introduce two novel criteria to establish an asymmetric sample selection and mining strategy based on the relationship between model predictions, focusing on their consensus and disagreement with given labels. Moreover, we propose a dynamic sample re-weighting method, utilizing historical training states to enhance the reliability of our clean sample selection and mining.(3) We present comprehensive experimental results on both synthetic and real-world datasets to demonstrate the superiority of our proposed ACT method. Moreover, we conduct extensive ablation studies to further validate the effectiveness of our approach. 
 
 Appendix:  


Decision: The paper, along with the assessments and the authors' rebuttal, has been thoroughly reviewed by the ACs. ACs also considered the confidential comments sent by the authors in the post-rebuttal discussions and their final decision. The authors introduce an asymmetric co-training method to tackle noisy labels in deep neural networks. While concerns, such as the lack of theoretical support, were raised initially, all reviewers have recommended acceptance after the rebuttal. The ACs concur with this decision and therefore recommend accepting the paper. Congratulations to the authors. Please carefully consider the reviewers' feedback and suggestions for revision, ensuring to address the promises made in the rebuttal.

Reviews: review: paper_summary: paper_summary: This paper proposes an Asymmetric Co-Training (ACT) method to enhance the robustness of deep neural networks (DNNs) when dealing with noisy labels. The ACT method involves training two models, the Robust Training Model (RTM) and the Noisy Training Model (NTM), using different strategies: RTM is trained with a subset of clean samples, while NTM is trained with the entire dataset. The method introduces novel sample selection criteria based on consistency and discrepancy between model predictions and labels, as well as a dynamic sample re-weighting mechanism that utilizes historical information. The effectiveness of the ACT method is demonstrated through experiments on both synthetic datasets (CIFAR100N and CIFAR80N) and real-world datasets (Web-Aircraft, Web-Bird, Web-Car, and Food-101N).
strength_weakness: strength_weakness: The paper proposes a novel Asymmetric Co-Training (ACT) method to enhance the robustness of models when dealing with noisy labels. The Robust Training Model (RTM) and the Noisy Training Model (NTM) are trained using different strategies: RTM with a subset of clean samples and NTM with the entire dataset. Additionally, the paper introduces new sample selection criteria based on the consistency and discrepancy of model predictions, as well as a dynamic sample re-weighting mechanism to improve the reliability of sample selection. These innovative strategies offer a new approach in the realm of co-training methods.
limitations: limitations: 1. Lack of Theoretical Support

- No Theoretical Proof: Although the paper proposes a novel method, it lacks theoretical validation. There is insufficient theoretical analysis to support the effectiveness of the proposed Asymmetric Co-Training (ACT) method.
- Insufficient Experiment Robustness: The experiments were not repeated multiple times, and standard deviations were not reported, making it difficult to assess the robustness of the results.

2. Poor Experimental Performance

- Limited Improvement on Real Datasets: The experimental results on real-world datasets show limited improvement, failing to fully demonstrate the advantages of the method.

3. Issues with Dataset Selection
- Unconventional Datasets: The datasets used in the experiments are not commonly used in this field, such as CIFAR-10, SVHN, or Clothing1M. Notably, CIFAR-100N generally refers to another real-world dataset (as seen in this paper: https://arxiv.org/pdf/2110.12088v2).
- Limited Noise Types: The authors should consider using more diverse types of noise, including pairflip and instance-dependent noise, to more comprehensively validate the method’s effectiveness.

4. Threshold Determination
- Insufficient Assumptions: The paper does not provide sufficient assumptions for the self-adaptive metric used to determine when NTM fits the noise. Additionally, experiments should be added to validate the effectiveness, enhancing the credibility of the method.
paper_summary: paper_summary: The paper suggests training two networks on different data to achieve asymmetric training. Among them, RTM trains on the selected clean samples, while NTM trains on the entire noisy training set. It also leverages historical training states and the consensus and disagreement between the two models to facilitate robust training.
strength_weakness: strength_weakness: [1] This paper presents everything very well. The layout of the figures and text is exquisite, the introduction of related work is detailed and thorough, and the presentation of the main ideas and experiments is also sufficiently satisfactory.
[2] This work is a simple but effective improvement to the strategy of co-teaching.
limitations: limitations: The experiments presented in this article are confusing. CIFAR-100N is the name of a natural noise dataset, but the authors use CIFAR100N in the text to refer to a synthetic noise dataset with a certain label noise rate. The authors should use a different term to describe this synthetic dataset to avoid confusion for readers. Moreover, I can confidently say that many methods used in Table 1 do not involve the evaluation baselines using CIFAR100N and CIFAR80N (perhaps the authors can point me to works that use these datasets to evaluate method performance. The authors claim they are following Jo-SRC, Jo-SRC claims it is following JoCoR, but JoCoR follows standard practices without using terms like 80N). CIFAR100N and CIFAR80N seem to be a very challenging synthetic version of CIFAR100. Under Sym-20% noise, SOP and DivideMix, two methods considered to have excellent performance in learning with label noise, can only achieve 50%+ performance. Table 2 does not appear to be a standard baseline test either. Notably, on Food101N, the authors' method achieves similar performance to DivideMix.
Although I agree that performance is a relatively unimportant factor in whether a proposed method should be recognized, as a reader, I expect a performance comparison on comparable and widely-recognized baselines to convince me of the effectiveness of the proposed method.
paper_summary: paper_summary: The paper addresses the problem of label noise in training datasets, which significantly degrades the performance of deep neural networks. Traditional symmetric co-training methods, where two models with identical architectures are trained together, often converge and become less effective in handling noisy labels. To mitigate this, the authors propose an Asymmetric Co-Training (ACT) approach. This method involves training two models: a Robustly Trained Model (RTM) using a clean subset of the data, and a Non-Robustly Trained Model (NTM) using the entire dataset. The ACT framework introduces novel criteria for sample selection and mining based on model agreement and disagreement, guiding the training process and dynamically re-weighting samples based on their reliability.
strength_weakness: strength_weakness: 1.This paper introduces a novel asymmetric co-training algorithm. By selecting consistently agreed-upon examples, the algorithm avoids overfitting to noisy examples.

2.The paper is well-organized, and the selected criteria are reasonable, supported by validation experiments.

3.The method significantly improves performance on a synthetic dataset, demonstrating the effectiveness of the proposed approach.
limitations: limitations: Weaknesses & Suggestions

1.The motivation is unclear. In the introduction, the authors suggest that the success of SCT mainly stems from distinct initializations. However, the success of ACT also heavily depends on distinct initializations to ensure the models are different enough to select clean examples. This similarity weakens the paper’s motivation.

2.The writing in Section 3.3 lacks clarity, with two main issues: 1) Figure 3 is difficult to understand because the x-axis is not defined, nor is the calculation of the metric explained. 2) The presentation is ambiguous; for instance, it's unclear whether scenario (2) mentioned in line 461 refers to the same situation as in line 430. As this subsection is central to the paper, it needs clearer organization and explanation.

3.The sensitivity analysis of key parameters is missing or incomplete. The number of warm-up epochs $E_w$ affects the training process: if too large, NTM starts to overfit noisy data, while if too small, the model agreement resembles a random result. The choice of this parameter also impacts the selection of $\tau$, making its analysis crucial. For $\tau$, sensitivity analysis in the Appendix indicates that most optimal values occur at $\tau=0.25$. Exploring what happens with $\tau < 0.25$ and its implications could be valuable. Moreover, the claim that the model is robust to changes in $\tau$ is questionable, as performance drops significantly in Sym-80%.

4.Compared to synthetic datasets, the proposed method shows limited improvement on real-world datasets. Exploring hyper-parameter selection might help extend the success from synthetic to real-world datasets.

5.It would be helpful to highlight the best competitor in the table.

Typos 

1.In Eq. (1), it would be clearer if $\theta$ were denoted as a subscript rather than an input of F, given that it is represented as a mapping from X to Y.

2.In statement (2), if it is characterized as empirical risk, then it should be associated with D.  Additionally, here you use a semicolon in F.

3.In the Appendix, line 153, change $\lambda$ and $\tau$. 

Although this paper has significant room for improvement, its novelty and potential are evident. Therefore, I rate it as borderline and am willing to raise my score following the rebuttal period.