Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this manuscript, the authors discussed the sensitivity of deeper and shallower layers in a DNN and improved the conventional early stop of the whole model to a layer-wise progressive early stop (PES) for label-noise learning.
main_review: The idea is interesting. The authors improved the conventional early stop method in the label noise learning task. However, I’m confused with the task setting. Furthermore, the proposed method is too naive and the authors did not provide key part of the proposed method (see the third point in Limitations And Societal Impact). The related work was adequately cited. The manuscript is well-written. The experimental results seem good on two small-scale datasets, but the authors did not evaluate their method on ImageNet, which is a common setting.

1. In figure 1, the shallower layers are much better than the final layer with much higher accuracies in each case. How to understand this statement? Meanwhile, compared with figure 2, the test accuracies of the 9th layer are much higher than those of the PSE. Should we say that we can directly use the outputs of the 9th layer of the ResNet18, rather than the final outputs? This determines the significance of the label noise learning task.
2. As in figure 1, the shallower layers converge later than the deeper layers, which means the shallower ones should train in more epochs. However, the authors first froze shallower layers in eq (4) and algorithm 1. They are the opposite. How to explain this?
3. The authors did not discuss how to select T_1 to T_L in section 2.1, which is truly important. This makes the proposed method naive and hard to follow.


limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents an interesting new method namely progressively early stopping to better exploit the memorization effect of DNN for learning with noisy labels. It first analyzes different effects of noisy labels to different DNN layers and finds that latter layers are much more sensitive to noisy labels. Based on this understanding, the authors then propose a progressively early stopping method, which progressively stops the training of different layers to counteract the influence of noisy labels. By combining some existing semi-supervised learning techniques, the proposed method achieves SoTA performance on several popular image classification benchmarks.
main_review: Pros:
-	Strong and clear motivation. The authors first analyze the influence of noisy labels on different DNN layers during different training iterations, which shows that latter layers are much more sensitive to noisy labels. Then, with this understanding, the authors propose to train different layers with different early stopping iterations to better exploit the memorization effect. The authors are the first to analyze the memorization effect for different DNN layers. 
-	Solid technical contribution. The authors first split a DNN into different parts and optimize each part with different training iterations. Since latter layers are verified to be more sensitive to label noise, by applying smaller training iterations, the obtained model can be less overfitted to noisy labels. To exploit this phenomenon, the authors propose a new method dubbed progressively early stopping, which is also demonstrated to be easily combined with other semi-supervised learning techniques. 
-	State of the art experimental performance. Their simple approach shows promising results and yields new SoTA performance on both synthetic and real-world datasets.
-	Convincing ablation experiments. The authors adopt a base CNN model and compare traditional early stopping with the proposed progressively early stopping. The results confirm that progressively early stopping is superior to the alternative. 
-	Presentation is clear and easy to follow. 

Cons: 
-	The authors claim that different DNN layers have different sensitivities to noisy labels. Fig.1 demonstrates this point by splitting the DNN into three parts, while in the experiments, the authors alternatively split the adopted DNN into two parts. According to the paper, how the network is split seems important to the proposed method but is not introduced in detail. 
-	In table 1, according to label precision, PES cannot surpass traditional early stopping, which seems not consistent with the claim of the method. And this issue is not explained in detail. Therefore, a better explanation is expected.
-	In Table 2, The proposed method seems always performs even better than other baselines when the noise is heavier under all types of noises. Authors should elaborate on this phenomenon to better understand the PES method.
limitations_and_societal_impact: No concern was found. This paper should have positive social impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper tackles the problem of classification under a noisy data setting. The authors make the observation that noisy labels mainly affect the latter layers of deep neural networks, and propose a method to address this, termed progressive early stopping (PES). A network is trained by freezing the parameters layers of layers 1,2,...,L (numbered from input to output) at predefined epochs E1, E2...EL. After this step, the authors use the network thus trained to identify “confident” training examples (those for which the predicted label matches the true label). Then the paper proposes two ways of training a final model: (1) in a fully supervised fashion, using only the confident examples (while accounting for class imbalance), or (2) semi-supervised, where the examples that are not confident are considered unlabeled. The approach is tested on noisy versions of CIFAR10 and CIFAR100 and the Clothing-1M dataset, and compared with competing supervised and semi-supervised methods.  

main_review: Overall, I found this paper interesting and well written. It proposes some interesting ideas and provides thorough comparisons with competing methods. The authors also addressed several issues that could be considered weaknesses such as training run time. On the negative side, I did not find the proposed “progressive early stopping” (PES) strategy very original, as it bears a close resemblance to greedy layer-wise training [1,2], and the authors do not discuss this similarity. However, even if this part is not novel, the authors do use it in what I think is a novel way to handle noisy labels. One aspect that I am a bit unclear about is whether the PES part is necessary for the second part of the paper starting from section 2.2.  -- what if you skip that part and compute the confidences in (5) using a normally trained baseline and then apply the proposed semi-supervised approach? Aside from this, the experimental analysis looks pretty good.

I will discuss this in more detail below.

**Originality:**  As mentioned above, the PES component is very similar in spirit to [1] and [2] (there are probably other papers too, but these are pretty popular), and while the methods may not be identical, without a comparison in the paper is hard to understand what is new. Using the predicted noisy labels as unlabeled data in semi-supervised learning is also not new, and it was proposed in DivideMix (cited as [15] by the authors). So in fact the combination of PES and the idea in DivideMix is what in fact makes this method work so well. 

[1] Bengio, Yoshua, et al. "Greedy layer-wise training of deep networks." Advances in neural information processing systems. 2007.

[2] Belilovsky, Eugene et al. “Greedy Layerwise Learning Can Scale to ImageNet.” ICML 2019

**Quality**: I really appreciated the experimental analysis in this paper, which considers many baselines, does a sensitivity analysis on the hyperparameters and reports run times. The paper contains a lot of details about hyperparameter configurations, although it’s not clear to me how these were set, especially the number of epochs T1, T2… which also differ per dataset. Could the authors please clarify this?

**Significance**: The tackled problem is definitely of considerable practical importance, and the authors report significant improvements over existing methods. I am not though familiar with the most recent approaches for noisy labels to assess if the competing methods are state-of-the-art; perhaps the other reviewers can address this.
The method does have some limitations that hinder its use in practice, such as longer training time especially for the semi-supervised version, and the fact that there are multiple stages to it. However, this may be a trade-off that some users may be willing to pay for the improved performance.

**Some clarification questions:**
1. Line 161: how can we make sure we have enough labeled samples? For a bad classifier with low accuracy, many of the samples will be put in the unlabeled set, to the point where you may have too little data to train a good classifier using semi-supervised learning. What is a minimum accuracy necessary for the baseline classifier (the one used to compute the \hat y) in order for this method to have enough labeled samples to perform better than the baseline itself?
2. How were the hyper-parameters T1, T2… set in the various experiments? What about the rest of the training hyperparameters, such as learning rate, regularization etc.? 
3. Eq. (5) Do the authors use only 2 random augmentations? Are two enough to get a reliable estimate? How sensitive is it to how much you alter the original image? 
4. For methods “CE” in Tables 2 and 3? Is it standard training with cross-entropy on the noisy dataset? If so, how do you make this semi-supervised.
5. In case this is not covered by the previous question, what happens if you compute \hat y_i with early stopping in eq. (5), and then apply your semi-supervised approach on this? Is this reported in the table?

limitations_and_societal_impact: Yes, the major limitations that I could think of are the running time, number of hyper parameters and sensitivity to hyper-parameters, which the authors have addressed. The other potential issue, I have mentioned in question 3 above.

I don’t think societal impact is discussed, but I don’t think this is a concern for this type of method and evaluation setting.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper introduces a novel training scheme (progressive early stopping, PES) aimed at improving the performance of convolutional networks for image classification in presence of noisy labels. The strategy is motivated by the observation that the layers close to the output tend to overfit noisy targets more than early layers. PES thus proposes to differentiate the number of training epochs for different groups of layers, training early groups for more epochs than final ones. In detail PES divides the networks in N blocks and trains for T_1 epochs the whole network. It then freezes the learned weights of the first block and  reinitializes  and trains the layers of the remaining N-1 blocks for T_2 epochs. Crucially, T_2 < T_1. The same policy is then progressively repeated for all the remaining N-2 blocks with T_n <T_n-1 for n in  {1, …, N}. 
PES is also combined with a semi-supervised scheme (MixMatch) to reach SOTA performances on CIFAR10 and CIFAR100 datasets.

main_review: Quality:
1)The training policy described in lines 192-196 is quite unclear. Here the number of epochs is set to 200 and the learning rate, starting from 0.1, is decreased by a factor 10 at epochs 100 and 150. Then T1 is set to 20/35, T2 to 7, T3 to 5 and the learning rate is set to 10**-4 at T2 and T3. What happens after T1+T2+T3 epochs? The value of the learning rate after the first T1 epochs is 0.1 or 10**-4?

2) What is the motivation for introducing two different optimizers, i.e. SGD with a step learning rate scheduler and Adam in the same training run (see lines 192-209)?

3) I have a concern regarding a comparison between the results reported in Table2 and the sensitivity analysis shown in Fig.3. If I understood correctly, PES with T2=0 and T3=0 should be equivalent to the standard cross entropy training since the whole network is trained for T1 epoch with noisy labels. From Table2  we can see that the accuracy gap between CE and PES for, say, cifar100 trained with symmetric noise 50% is between 16/21%. From Figure 3 instead we see that the accuracy gap between the optimal T2/T3 and T2=T3=0 is well below 10%.  A similar concern regards the other perturbations shown in the sensitivity analysis.
Why is this happening? 

4) As correctly recognized by the author(s) the approach requires fixing several metaparameters, in particular the number of epochs for each sub-block optimization. But why is this necessary? The common practice in early stop is finishing the optimization close to the maximum of the  accuracy estimated on a small dataset not used for the optimization (and different from the validation set). Why cannot this procedure be used in PES?

5) The code used to train the network should be publicly shared.

Clarity:

The paper could be improved in the following parts:

When the authors describe Fig. 1 they should say on which dataset they train the network used to produce the profiles, (only looking at table 3 of section 3.2 it is possible to guess that the dataset is Cifar10), and for how many epochs they trained the first block of the network. Moreover the meaning of ‘clean data’ line 53 is not very clear. Indeed if one compares the accuracy of a classifier trained only on a noisy  dataset with that obtained with the same classifier on noisy and noiseless data it seems quite natural that in the second case the accuracy improves;

Line 172: it is not clear what ‘adiacent class’ means;

There are a few typos and grammar errors.

limitations_and_societal_impact: The limitations are rather briefly but rather clearly discussed at the end of the paper.
review_title: null

Meta-review: The paper proposes a variant of early stopping for learning with noisy labels. The method starts with the hypothesis that early layers are less prone to be adversely affected by label noise than later layers. Authors empirically validate this hypothesis and then propose a progressive layerwise training method for tackling label noise. This is then used to identify noisy examples which are treated as unlabeled data and existing semi-supervised methods are used to get further empirical gains. Reviewers have expressed some concerns on the heuristic components in the method (use of Adam for last layers vs SGD for earlier layers), several metaparameters (the number of epochs for each subblock), and the increased training time of the method due to progressive/multi-stage training, but they have found the motivation behind the paper convincing and have been overall positive about the paper. I believe the paper is above the acceptance threshold for the publication. 
Decision: Accept (Poster)