N-Student Learning: An Approach to Model Uncertainty and Combat Overfitting
Authors: Ryan Y. Xu
Solomon E Garber
Antonella Di Lillo
James Storer
Venue: Submitted to ICLR 2023 (2023)
Keywords: Noisy Labels
Pseudo-Labels
Overfitting
Supervised Learning

Abstract: This work presents N-Student Learning, a pseudo-label based multi-network training setup that can be applied to nearly any supervised learning architecture in order to help combat the problem of overfitting and control the way in which a network models uncertainty in the data. The effectiveness of N-Student Learning relies on the idea that a network's predictions on unseen data are largely independent of any instance-dependent noise in the labels. In N-Student Learning, each student network is assigned a subset of the training dataset such that no data point is in every student's training subset. Unbiased pseudo-labels can thus be generated for every data point in the training set by taking the predictions of appropriate student networks. Training on these unbiased pseudo-labels minimizes the extent to which each network overfits to instance-dependent noise in the data. Furthermore, based on prior knowledge of the domain, we can control how the networks learn to model uncertainty that is present in the dataset by adjusting the way that pseudo-labels are generated. While this method is largely inspired by the general problem of overfitting, a natural application is found in the problem of classification with noisy labels — a domain where overfitting is a significant concern. After developing intuition through a toy classification task, we proceed to demonstrate that N-Student Learning performs favorably on benchmark datasets when compared to state-of-the-art methods in the problem of classification with noisy labels.

Under review as a conference paper at ICLR 2023
N-STUDENT LEARNING: AN APPROACH TO MODEL UNCERTAINTY AND COMBAT OVERFITTING
Anonymous authors Paper under double-blind review
ABSTRACT
This work presents N-Student Learning. pseudo-label based multi- network train- ing setup that can be applied to nearly any supervised learning architecture in order to help combat the problem of overfitting and control the way in which network models uncertainty in the data. The effectiveness of N-Student I Learning relies on the idea that network's predictions any instance-dependent noise in the labels. In N-Student Learning, each student network assigned subset of the training dataset such that data point is in every student's 's training subset. Unbiased pseudo-labels can thus be generated for every data point the training set by taking the predictions of appropriate stu- dent networks. Training these unbiased pseudo-labels minimizes the extent to which each network overfits to instance-dependent noise in the data. Furthermore, based on prior knowledge of the domain, we can control how the networks learn model uncertainty that is present in the dataset by adjusting the way that pscudo- labels are generated. While this method is largely inspired by the general problem of overfitting, natural application is found in the problem of classification with noisy labels domain where overfitting is significant concern. After devel- oping intuition through a toy classification task, we proceed to demonstrate that N-Student Learning performs favorably on benchmark datasets when compared state-of- oftte-of-he-aadd art methods in the problem of classification with noisy labels.
unseen data are largely independent of
INTRODUCTION
Overfitting is a fundamental problem in supervised classification which a model learns properties of the training data that do not generalize to unseen data. If samples from the input space contain information that is not relevant to the task, the model may overfit by learning to use these irrelevant details for predictive purposes. Overfitting may also occur as a result of noise in the label space, in which the label provided in the dataset does not match the expected output of the model given the corresponding input. Overfitting due label noise in its various forms will be a primary focus of this paper.
In this paper, we introduce N-Student Learning, a pseudo-label based multi-network training setup which mitigates overfitting to noise in the labels. The idea is to relabel the dataset by taking the predictions of networks that have never seen the data. We do this by training multiple networks on different subsets of the data so that the pseudo-labels that they generate on their respective unseen subsets will be clean of any instance-dependent noise. Training on these pseudo-labels results in networks that are less prone to overfitting.
After introducing the architecture section 2 we will discuss a few types of label noise that are commonly present in datasets. Using toy classification problem in section 3 we show the effect of the N-Student Learning setup and demonstrate that the setup can be adapted to handle different kinds of noise. Following this, in section 4 show that N-Student Learning performs favorably when compared to state-of- -f- methods on both artificially noisy and naturally noisy benchmark datasets.

---

Under review as a conference paper at ICLR 2023
2 METHOD
2.1 TRAINING SETUP
Supervised classification is the problem of learning a function that maps an input space X to label space Y, given a dataset D { Vo H where 2; E X and Ye E Y, and is the S is the size of the dataset.
In N-Student Learning, we train n student networks {N.) in parallel, where each network N; trains on a subset D; of the dataset. In our work, is assumed that these networks all have the same architecture. These subsets are generated with the constraint that for each sample (Ii, Us) in the dataset, there is some network that does not train on that sample. This allows us to generate clean pseudo-labels for each sample. Thus, the following must hold:
D U (E(1,2,.....)
Di, where e denotes a set complement.
We begin training with a warmup phase where we train each N₂ on its respective dataset D; for a small number of epochs without pseudo-labels. After the warmup phase, each student continues to train on its own subset D. but with portion of the labels replaced by pscudo-labels generated by a student that has never seen the respective inputs. We define a hyperparameter p. the pseudo-label rate, which controls what proportion of the ground-truth labels will be replaced by pseudo-labels At the beginning of each epoch, we randomly select a subset saampl of size [pS]. For each pair {zi, We E Dample we replace it with { Nj (z:)), where Nj is a network that does not train on æ; and NJ (at)) is pseudo-label generated from the prediction of Nj 24. After this pseudo-labeled dataset is generated, we train each student on its respective pseulo-labeled subset.
We experiment with three methods for generating pseudo-labels: hard, soft, and stochastic pseudo- labels. Hard pseudo-labels are generated by taking the argmax of the predicted logits and assigning generated by
that class as the correct label in the form of one-hot vector. Soft pseudo-labeli using the distribution defined by the output of the softmax layer as the target vector. Lastly, we define stochastic pseudo-label, to be pseudo-labels generated by sampling from the distribution output by the network and representing the sampled class as a one-hot vector.
We restrict our experiments to cases in which each student is allocated training subset of equal size. For 2 students, this means that each student trains on half of the data, and therefore generates pseudo-labels for the other half. For 3 students, each student can train on anywhere from one third to two thirds of the data with this equal-allocation restriction. To further constrain the choice of subsets, we only consider splits in which the fraction of the dataset per student is on the boundary of the possible range of values. For n students, this means that each student trains on either or n-1 of the data. This is because if each student trains on less than 1 of the dataset, then part of the dataset is not used for training because UUEE(.......)) Dj| & n Dol < [D]. If each student trains on more than n-1 then each student can pseudo-label less than 1 of the dataset, which is not enough to label the whole dataset.
In the case that each student trains on of the data, the training subsets disjoint and for any student's subset, each of the n - other networks are able to pseudo-label the samples from that subset. Each epoch, we choose random network to pseudo-label these samples. If each student trains on p-1 of the data, then there is only one network that can pseudo-label each sample, so there are no choices to be made.
There are a variety of ways to perform inference when training multiple networks. One way is to choose a random network after training to use for inference. Alternatively, predictions can be aggre- gated by averaging or multiplying class-wise before normalizing, which can improve performance.
Important hyperparameters: Number of students Pr Pseudo-label rate (proportion of labels that replaced by pseudo-lahes lahels each epoch)
2

---

Under review as a conference paper at ICLR 2023
3 REPRESENTING UNCERTAINTY
Although the goal of supervised classification is to predict correct labels, we often want classifier whose predictions reflect the correct conditional probability distribution over the labels given an input sample- not just the highest likelihood class. In the following paragraphs, we will describe few types of classification problems and how they may impact the training of an effective classifier.
Consider problem in which a label can determined with near certainty for any given input sample. An example of such a domain may be the classification of unambiguous high-resolution images. In this case, learning the distribution of the dataset and fitting to the dataset's particular samples may be sufficient to effectively model the problem.
An interesting case arises when working with domains which contain inherent stochasticity. An example such a case is a future prediction problem, such as the problem of predicting whether house will be sold using only information about features of the house. Given information about any particular house, you can never have absolute certainty that the house will be sold or not. Despite this uncertainty, the dataset that we provided contains binary labels representing whether particular house has been sold. The main distinction between this case and the last is that in the stochastic problem, each instance of data in the training may not be representative of the correct distribution that you want to learn. However, similar to the previous unambiguous case, the dataset as a whole may be representative of the correct distribution of the problem.
Finally, consider the case of an unambiguous problem that simply contains errors in the labels. In this case, both the individual samples and the dataset as whole may not fully representative of the task. We explicitly do not want to model the exact distribution that is found in the data. Instead, we can ignore these errors by modeling the most likely class label, filtering out any uncertainty. This is also precisely the problem of training with noisy labels.
key insight is that without prior knowledge about the domain and the way that the dataset was generated, these three cases may be completely indistinguishable. Despite this, we may want to represent uncertainty that is present in these datasets in different ways depending on the type of noise that is present. In section 3.1 we will show that based the priors that one has dataset, N-Student Learning can be used to appropriately model the uncertainty the data by adjusting the way that pseudo-labels generated.
3.1 A ToY PROBLEM: CLASSIFICATION OF TWO GAUSSIANS
In this section, we demonstrate the impact of N-Student Learning on overfitting through a toy clas- sification problem. We also show how the choice of pseudo-labeling method impacts the network's 's outputs. The approaches will be compared by visualizing network predictions across the 2-d input space.
3.1.1 TWO GAUSSIANS: EXPERIMENTAL DESIGN
The dataset for this classification problem is generated by sampling 1000 total points from two circular Gaussians separated by 2 units, each with unit variance. In fig. we see the data sample.
The networks that we use for the baseline and for the Student setups are standard feed forward networks with 5 hidden layers of 100 neurons and ReL.U activation. They trained for 400 epochs with stochastic gradient de- scent with 0.9 momentum and batch size 16.
In order to demonstrate the effect of each training setup, we visualize the output of the network by displaying the predicted class probabilities for each point the input space. These probabilities are represented using a gradi- ent between blue and yellow in fig. and fig.
Figure 1: A dataset of 1000 total points generated by sampling from two circu- lar Gaussians.
3

---

Under review as a conference paper at ICLR 2023
1.0
0.8
0.6
0.4
0.2
0.0
(a)
(b)
(c)
Figure 2: (a) shows the conditional probability that point is drawn from each Gaussian. (b) shows the most likely class given any data point. (c) shows the mapping between color and class probabil- ity.
3.1.2 Two GAUSSIANS: RESULTS
We see in fig. that the baseline approach overfits to the training set by epoch 400. In contrast, the 2-Student setups are more robust to overfitting.
Soft and stochastic pseudo-labels generate predictions that are fairly representative of the actual conditional probability that a point belongs to each Gaussian which be found in fig. (a). Hard pseudo-labed form sharp decision boundary down the middle, more akin to fig. (b).
10
100
Epochs
200
400
Baseline
Soft
Stochastic
Hard
Figure 3: The outputs of a baseline network compared to those of 2-Student setups trained using soft, stochastic, and hard pscudo-labels. The outputs are visualized at epochs 10, 100, 200, and 400. The color map is the same fig.
4

---

Under review as a conference paper at ICLR 2023
3.1.3 Two GAUSSIANS: DISCUSSION
Without prior on how the dataset is generated, you cannot distinguish between different explana- tions for the uncertainty that appears on the boundary between the two Gaussians. this case, the true generation process is fundamentally stochastic and the uncertainty in the middle results from a stochastic sampling process. However, it is also plausible that the two classes modeled in the dataset are completely disjoint in nature, separated by clear-cut decision boundary. In this case, the uncertainty may be explained by errors label generation that are more likely to occur towards the boundary. The optimal network outputs for these cases are shown in fig. 2
useful feature of N-Student Learning is that one can use their prior on the type of uncertainty that is present in dataset choose labeligg method that appropriately represents the uncertainty. If the problem is known to be fundamentally stochastic and minimally noisy, then we want to use soft stochastic pseudo-labels in order effectively model the conditional distribution over the classes given a data point. On the other hand, if we have reason to think that the dataset contains a lot of errors in the labels, perhaps as a result of the labeling process, then we can effectively filter out the errors near the decision boundary by using hard pseudo-labels.
4 NOISY LABELS
The task of classifying with noisy labels has been widely researched and has many applications. In addition to being a great task to study the problem of overfitting, learning train with noisy labels also has the potential to reduce our reliance on clean, human-labeled datasets. There is a trade- off between cost and accuracy when creating the datasets that we train on. At one end, have expensive carefully human-labeled datasets and on the other, we have inexpensive automatically generated datasets full of noise. Properly leveraging the latter efficient and scalable datasets requires training techniques that are more robust to noise in the labels.
4.1
RELATED WORKS
4.1.1 SAMPLE SELECTION METHODS
A popular class of methods for training on noisy data involves sample selection, technique in which clean instances of data are selected by relying on the "low-loss criterion". the principal that when a trained network predicts on the training set, lower loss samples are more likely to have clean labels. Below, we discuss a number of sample selection related approaches.
MentorNet is an approach that connects the problem of noisy data to curriculum learning (Jang et al., 2018). a sub-field that looks to generate curriculum for networks to train on. MentorNet first trains a teacher network on noisy data. This teacher network then used to generate a curriculum for a student network to learn off a dynamic weighting of the importance of the samples in the dataset.
Decoupling trains two networks in parallel with "Update by Disagreement" algorithm in which the networks are trained on the samples that the two networks disagree on (Malach & Shalev-
Shwartz 2017).
Co-teaching similarly adopts a two-network setup Han et al. 2018). Each epoch, the low-loss samples from each network are chosen to train the other network.
Co-teaching+ combines the Update by Disagreement algorithm from Decoupling and the low-loss criterion YYu et al. 2019). Accordingly, the networks train on samples where the networks disagree and have small loss.
JoCoR trains two networks in parallel on low-loss samples where the predictions of the two net- works align (Wei et al., 2020). The idea is that a sample is more likely to be clean if the networks agree in addition to the samples having low loss.
5

---

Under review as a conference paper at ICLR 2023
4.1.2 SEMI-SUPERVIE AND
METHODS
A class of methods reframes the noisy label problem a semi-supervised learning problem in which low-loss samples are used to split the dataset into a clean and unlabeled dataset in order to leverage existing semi-supervised learning methods. Self-supervised methods also particularly well-suited for the noisy-label problem because self-supervised pre- teeettriinng is not impacted by noise in the labels.
Div is a semi-supervised method which trains two networks et al. 2020). Every epoch, semi- cumi-supereieed methods used on the labeled-unlabeled split generated by the other network. The supervvpe methods used by DivideMix include techniques such as generating pseudo- labels by averaging and sharpening the predictions of network over multiple augmentations. The mixup algorithm is also used, which encourages linear behavior in a network by training on synthe- sized interpolated training examples that are generated by taking a linear combination of both the input images and the labels.
Co-learning is a self-supervised approach that takes further advantage of multiple augmentations for training (Tan et al. (021). A weakly augmented view and two strongly augmented views of the input are generated and passed through a shared encoder, where the encoded weakly augmented view is passed through classification head for traditional classification loss. The two strongly augmented views are passed through a projection head to a space where contrastive loss is applied, creating stronger encoder for the classification head to use. Finally, a loss is added that encourages the property that for any two given samples, the classifier head and the projection head's encodings have a similar distance. All of these properties work together to create robust classifier.
4.2 EVALUATION
Noisy label methods commonly evaluated both artificially corrupted datasets and datasets with natural noise. We will be evaluating on three artificially corrupted datasets: MNIST, CIFAR- 10, and CIFAR-100, well one naturally corrupted dataset, ANIMAL-IOn (Song al. 2019).
4.2.1 ARTIFICIALLY NOISY DATASETS
Artificially corrupted datasets are supervised classification benchmark datasets that have noise added to them. Two common types of noise injection methods are symmetric noise and asymmetric noise.
Symmetric noise generated by randomly corrupting each label into different label with some probability. As an example, symmetric noise with flip rate of 0.4 means that every data point's label will be flipped to a different label with 40% probability.
Asymmetric noise is designed to mimic class-dependent noise, in which one class may get confused with another. This can be implemented in a number of ways. For MNIST and CIFAR-100, past works implement asymmetric noise by corrupting each class into the next class with some prob- ability. Following previous works, "next" is arbitrarily chosen to be the class represented by the following index in the target class vector.
For asymmetric noise on CIFAR-10, past works corrupt the classes in a more meaningful way to mimic real noise (Arpit et al. 2017). Specifically, we have the following mappings: { bird + airplane, cat - dog, deer + horse, truck + automobile }o Note that only 4 of the 10 classes are corrupted here, so in the case of CIFAR-IO, the effective noise two fifths of what is typically expected for a particular noise level.
4.2.2 NATURALLY NOISY DATASETS
While artificially corrupted datasets are good domain for experimentation, would also like to perform well on datasets that naturally have noise, as these will be the most reflective of real-world datasets.
We report results on ANIMAL-10n, a dataset that contains pairs of confusing animals that are human labeled with an estimated noise level of 8%.

---

approaches, our results take the predictions of a single student in Under review as a conference paper at ICLR 2023
4.3 COMPARISONS
4.3.1 CIFAR-10, CIFAR-100
On popular benchmarks such as CIFAR-10 and CIFAR-100, we do not compare to methods for which the baseline performance (without added noise) is significantly stronger than baseline net- work trained with cross-entropy loss. These are typically methods that make use of strong aug- mentations and contrastive learning.
The approaches that we compare to are the following: Baseline, Decoupling, Co-teaching, Co- teaching+, and JoCoR.
Although do not compare to Co-learning due its use of contrastive learning, we choose to include Co-learning in table 2 and table 3 as a reference due to the relative low computational complexity compared other methods that we omit.
Following previous 2-Student architecture.
4.3.2 ANIMAL-ION
We compare to all public results ANIMAL-IOn. These include the following: Baseline, Co- Learning, Nested Co- Te-ahinnn (Chen et al. 202I), PLC (Zhang et al. 2021), SELFIE (Song et al., 2019), and S3 (enng et al. 2022)
Many of these methods are compute heavy, and so we train a 4-Student setup with multiplicative- aggregate predictions.
4.4 EXPERIMENTAL SETUP
We use the same architectures as previous works. For CIFAR-I0 and CIFAR-100, we follow Co- learning. For ANIMAL-10n, follow SELFIE. Details can be found in table T
Table 1: Training Setups
Dataset
CIFAR- 10, CIFAR-100
ANIMAL-10n
Architecture
ResNet-18
VGG-19
Warm-up Epochs
10
10
Epochs
200
100
Optimizer
Adam
SGD, momentum
Learning Rate
.001. linear decay from epoch 80
.01, linear decay from epoch 40
Augmentations
Weak (Rotation, translation)
Weak (Rotation, translation)

---

Under review as a conference paper at ICLR 2023
Table CIFAR-IO Results (% Accuracy)
Noise Type/Rate
Methods
Symmetric 20%
Symmefrie 50%
Symmetric 80%
Asymmetric 40%
Baseline
84.81
61.49
28.98
76.30
Decoupling
85.75
61.93
27.23
74.97
Co-teaching
90.29
63.45
28.03
74.25
Co-tcaching+
88.63
76.27
30.37
81.25
JoCoR
90.43
66,00
29.19
73,95
*Co-learning
92.21
84.49
61.20
81.42
2-Students
86.22
81.29
24.77
88.59
Table 3: CIFAR-100 Results (% Accuracy)
Noise Type/Rate
Methods
Symmetric 20%
Symmetric 50%
Symmetric 80%
Asymmetric 40%
Baseline
57.79
33.75
8.64
42.49
Decoupling
56.18
31.58
7.71
41.51
Co-teaching
64.28
32.62
6.65
40.62
Co-teaching+
55.40
26.49
8.57
38.98
JoCoR
62.29
30.19
6.84
39.72
*Co-learning
66.58
54.54
35.45
41.96
2-Students
53.58
40.58
15.30
48.65
4.5
RESULTS
4.5.1 CIFAR-10
In table 2 we see that on the 2-Student setup performs comparably to other methods on 20% sym- metric noise, outperforms other methods on 50% symmetric noise, but underperforms 80% sym- metric noise. Notably, the 2-Student setup outperforms all methods by a large margin on asymmetric noise with 40% noise, including Co-learning.
4.5.2 CIFAR-100
In table 3 see that our method outperforms all other methods in every metric other than 20% symmetric noise, where we underperform. Again, we find that the 2-Student setup particularly strong in the domain of asymmetric noise, where outperform all other methods.
4.5.3 ANIMAL-1ON
Table 4: ANIMAL--10n Results (% Accuracy)
In table we that our method outper- forms all methods other than S3, a method with remarkably strong baselines due to self- supervised learning.
Methods
Accuracy
Baseline
79,40
2 Student Setup
81.22
SELFIE
81,80
We also that the performance of N-Student Learning improves as we increase the number of students, demonstrated in section 4.5.4
Co-Learning 3-Student Setup
82.18
83,70
PLC
83,40
Nested Co- Co-Teaching
84.10
4- 4-Student Setup
84.57
S3**
88.50
* indicates method that do compare to discussed in section 4.3.1
** indicates pre-print the time of writing
8

---

Under review as a conference paper at ICLR 2023
Table Ablation Over Hyperparameters on CIFAR-I0 (% Accuracy)
Subset Proportion
Single Student
Aggregate (Multiplicatiiv)
I/2
81.29
82.27
1 1/3
75.66
82.74
2 2/3
82.04
83.98
1 1/4
72.38
82.25
4
3/4
82.20
84.72
4.5.4 VARYING
In this experiment, we show how the performance of N-Student Learning is affected by the number of students and the amount of data that each student trains and the inference method. We test on CIFAR-10 with 50% noise and p = 0.5. We vary n from to 4 and test the upper and lower bounds of the range of subset sizes that student can train on: n=1
The results can be found in table 5 Unsurprisingly, we find that aggregating the predictions of students is stronger than any isolated student's performance. For = 3 and 4, find that training each student on a larger subset performs better, the best performance achieved is with students, each training on of the data, reaching validation accuracy of 84.72%, which is higher than Co-learning.
4.6 DISCUSSION
In our results, we find that N-Student Learning performs particularly well on the ANIMAL-IOn dataset as well as on the asymmetric noise tasks. The strong performance on asymmetric noise is important because real noise commonly contains similar class-dependant asymmetries.
The area that N-Student Learning consistently underperforms in is 20% symmetric noise. We hy- pothesize that this is due to the lower baseline performance of the 2-Student setup due to the data limitation per student. This may be less of a concern when training on large real-world datasets, where even half of the dataset contains enough samples to effectively learn the task.
We find that this simple multi-network architecture outperforms state-of-the-arr approaches on benchmark noisy datasets settings that are most similar to natural noise. Our results demon- strate that filtering samples by low-loss criterion is not necessary for good performance on noisy labels.
5 CONCLUSION AND FUTURE WORKS
N-Student Learning is a flexible training setup that allows for control over the way that network learns model uncertainty that is present in the data. It is meta-architecture that can be applied on top of essentially any supervised learning architecture to allow for robust training in the face of overfitting.
The training setup of N-Student Learning can be modified in various ways that may yield superior performance. Notably, the architecture the size of the training subsets each student are held constant for each experiment. However, it seems likely that other setups may be effective. As an example, consider training a strong student with a more powerful architecture and a larger subset of the data, supported by set of weaker students cleaning the labels for the stronger student.
N-Student Learning can potentially be applied to tasks such as semi-supervised learning. In semi- supervised learning, in addition to splitting the labeled training set, the unlabeled portion of the dataset could be split as well. This could help minimize celf-confirmation bias and prevent overfit- ting the limited number of labeled samples.
is our hope that the discussion on label uncertainty and the ideas behind N-Student Learning will inspire further work variety of problems.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. closer look at memorization in deep networks. In International conference on machine learning, Pp- 233-242, PMLR, 2017.
Yingyi Chen, Xi Shen, Shell Xu Hu, and Johan AK Suykens. Boosting co-teaching with compres sion regularization for label noise. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2688-2692, 2021.
Chen Feng, Georgios Tzimiropoulos, and loannis Patras. S3: Supervised self-supervised learning under label noise, 2022. URL htkp/ttttps:/opeeeevvieee wwt.
http://ppnnneveeeeeeeefrerrrrrrr wtwpummmiidiie 11 9FYBeFG
Bo Han, Quanming Yao. Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31, 2018.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentoret: Learning data- driven curriculum for very deep neural networks corrupted labels. In International conference on machine learning. pp. 2304-2313. PMLR, 2018.
Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. 2020.
Eran Malach and Shai Shalev-Shwartz. Decoupling" when update" from" how to update". Ad- vances in neural information processing systems, 30, 2017.
Hwanjun Song, Minscok Kim, and Jac-Gil Lee. Selfie: Refurbishing unclean samples for robust deep learning. International Conference on Machine Learning, pp. 5907-5915. PMLR, 2019.
Cheng Tan, Jun Xia, Lirong Wu, and Stan Li. Co-learning: Learning from noisy labels with self-supervision. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 1405-1413, 2021.
Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13726-13735, 2020.
Xingrui Yu, Bo Han, Jiangehao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does dis- agreement help generalization against label corruption? In International Conference on Machine Learning, pp. 7164-7173, 2019.
Yikai Zhang, Songzhu Zheng, Pengxiang Wu, Mayank Goswami, and Chao Chen. Learning with Srature-dependee label noise: A progressive approach. In International Conference on Learning Representations, 2021.
10


Reviews:
[{"review_id": "gK1Hqia-vU7", "reviewer": "Reviewer_Riap", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an interesting \"N-student learning\" framework that can be used to tackle the framework of learning with noisy labels. Specifically, the method involves training multiple neural networks, each seeing only a non-overlapping subset of the entire training dataset. The authors of the paper also take one step further by adopting the use of pseudo-labels to further make the training more effective. They demonstrate that this simple method can achieve very good performance on several benchmark datasets tested. ", "summary_of_the_review": "All in all, due to the weaknesses mentioned above, I recommend rejecting the paper for now. ", "strength_and_weaknesses": "Strength: \n- Overall, the paper is well-written and easy to follow.\n- The proposed method is technically sound.\n- The experiments conducted are reasonable, and the proposed method can achieve good performance. \n\nWeaknesses:\n- My main concern regarding the paper is the lack of novelty. The use of pseudo-labels has been previously demonstrated to be effective to combat the issue of noisy labels. The proposed method of using multiple models each seeing only a subset of the entire training set also seems to be a rather straightforward idea. The main contribution of the paper is arguably the experimental comparison against several previously proposed methods for learning with noisy labels. I am not sure if this would be technically novel enough to be published. It might be more interesting if the authors of the paper can also offer some theoretical insights on the effect of multiple networks each one only seeing a subset of the training dataset. \n- Another concern I have is the lack of an ablation study. For instance, what if we had the same number of models, but each trained using the entire dataset? Would that be worse than the proposed method? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "eUgmn6GxbLB", "reviewer": "Reviewer_X44G", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an ensemble based label-propagation method for classification in the presence of label noise. In the method, the authors emphasize the importance of generating pseudo-labels only for datapoints a model has not seen during training. Empirical comparisons against existing algorithms are limited but promising.", "summary_of_the_review": "The proposed method is quite simple and could be promising, but I believe this paper in its current format is not fully convincing and lacks rigor and scientific significance. I find this to be mostly an empirical work that seems also rather light on the empirical findings and not extensive on the range of experiments and ablations performed. I would like to encourage the authors to build on this with either more analysis or empirical studies.", "strength_and_weaknesses": "Strength:\n- Writing is clear overall and readability is good.\n- Empirical results show promise of method.\n\nWeakness:\n- Analysis and explanation of method is lacking.\n    - The empirical results are promising, but there is little to no intuition for why this method works.\n    - E.g. while the authors stress the importance of pseudo-labelling only for datapoints a network has never seen, and that this produces \"clean\" labels which are \"clean of any instance-dependent noise\", there is little to no discussion/reasoning/analysis on why this is so significant. At the very least, there should be a discussion/ablations on the  of the pitfalls of pseudo-labelling for previously seen datapoints.\n    - Further, the proposed method seems to depend on each network successfully learning general representations of the data, such that the pseudo-labels reflect the underlying structure in the data. What mechanisms do you have that will enforce this will occur when each network is still learning with noisy labels? Or is the performance of the proposed method simply an empirical finding?\n- Details and rigor lacking overall.\n    - In all of the experiments, what is the \"baseline\" method?\n    - Are all of the results reported for just one seed? No repeated trials and standard errors?\n    - The authors mention 3 ways of generating pseudo-labels: hard, soft, stochastic. Which labeling method was used for the evaluations in Section 4.2?\n    - What is the exact formula to multiply predictions to aggregate them during inference? Do you multiply class probabilities, then normalize the output again into a probability? If so, what is the rationale for such an equation?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OUeMxq1ptO", "reviewer": "Reviewer_JspX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes N-student learning to combat overfitting in neural network training such as that caused by label noise. The proposed method trains multiple neural models at the same time in a cross-fitting manner. Namely, each model is trained only on a subset of the training examples, where some of the training labels that are likely to be noisy are replaced with the predictions of other models. The authors experiment on multiple datasets to showcase their method. ", "summary_of_the_review": "As noted above, this paper lacks novelty and its quality and clarity also need to be improved. Significant revision is required to make it ready for publication.", "strength_and_weaknesses": "Strength: \n* The paper is overall organized well and easy to follow. \n\n\n\nWeakness:\n* The paper is lacking novelty. Leveraging the cooperation between multiple networks to combat overfitting and label noise is not new. There exists a handful of works following this vein such as Co-teaching, DivideMix, and Co-learning, as also mentioned by the authors in the related work section. However, there is no detailed discussion of the differences between the proposed method and existing methods. \n\n* The quality of the work needs to be improved. Since the proposed method is claimed to be generally applicable and is similar to existing methods, the only way to demonstrate its advantage is through experimental results. However, based on the experiments, the proposed method is not significantly better than the existing methods. Sometimes it may be even worse than vanilla training without any techniques designed for label noise (e.g. Table 2).  Furthermore, some existing methods are missing in the experimental results, such as DivideMix, which I believe is a reasonable baseline without employing self-supervised learning.\n\n\n\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject