Early Stopping Against Label Noise Without Validation Data
Authors: Suqin Yuan
Lei Feng
Tongliang Liu
Venue: ICLR 2024 poster (2024)
Keywords: label noise
early stopping
deep learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose a novel early stopping method that tracks prediction fluctuations on the training set to select the desired model without any hold-out validation data, in the presence of label noise.

Abstract: Early stopping methods in deep learning face the challenge of balancing the volume of training and validation data, especially in the presence of label noise. Concretely, sparing more data for validation from training data would limit the performance of the learned model, yet insufficient validation data could result in a sub-optimal selection of the desired model. In this paper, we propose a novel early stopping method called Label Wave, which does not require validation data for selecting the desired model in the presence of label noise. It works by tracking the changes in the model's predictions on the training set during the training process, aiming to halt training before the model unduly fits mislabeled data. This method is empirically supported by our observation that minimum fluctuations in predictions typically occur at the training epoch before the model excessively fits mislabeled data. Through extensive experiments, we show both the effectiveness of the Label Wave method across various settings and its capability to enhance the performance of existing methods for learning with noisy labels.

Published as a conference paper at ICLR 2024
EARLY STOPPING AGAINST LABEL NOISE WITHOUT VALIDATION DATA
Suqin Yuan' Lei Lei Feng? Tongliang Liu'
Sydney AI Centre, School of Computer Science, The University of Sydney 2 School of Computer Science and Engineering, Nanyang Technological University
ABSTRACT
Early stopping methods in deep learning face the challenge of balancing the vol- ume of training and validation data, especially in the presence of label noise. Concretely, sparing more data for validation from training data would limit the performance of the learned model, yet insufficient validation data could result a sub-optimal selection of the desired model. In this paper, we propose a novel early stopping method called Label Wave, which does not require validation data for selecting the desired model in the presence of label noise. works by track- ing the changes in the model's 's predictions on the training set during the training process, aiming to halt training before the model unduly fits mislabeled data. This method is empirically supported by our observation that minimum fluctuations in predictions typically occur the training epoch before the model excessively fits mislabeled data. Through extensive experiments, of the Label Wave method across various settings and its capability to enhance the performance of existing methods for learning with noisy labels.
show both the effectiveness
INTRODUCTION
Deep Neural Networks (DNNs) are praised for their remarkable expressive power, which allows them to uncover intricate patterns in high-dimennional data (Montufar et al., 2014; LeCun et al., 2015) and can even fit data with random labels. However, this strength, often termed Memorization (Zhang al., 2017), label noise exists, the inherent capability of DNNs might cause the model to fit mislabeled examples from noisy datasets, which can deteriorate its generalization performance. Specifically, when DNNs are trained on noisy datasets containing both clean and mislabeled examples, it is often observed that the test error initially decreases and subsequently increases. To prevent DNNs from overconfidently learning from mislabeled examples, many existing methods for learning with noisy labels (Xia et al., 2019; Han et al., 2020; Song et al., 2022; Huang et al., 2023) explicitly or implicitly adopted the operation of halting training before the test error increases a strategy termed "early stopping".
be a double-edged sword, especially when encountering label noise. When
Early stopping relies on model selection, aiming to choose a model that aligns most closely with the true concept from range of candidate models obtained during the training process (Mohri et al., 2018; Bai et al., 2021). I). To this end, leveraging hold-out validation data to pinpoint an appropriate early stopping point for model selection becomes prevalent approach (Xu & Goodacre, 2018) in deep learning. However, this approach heavily relies on additional validation data that is usually derived by splitting the training set, thereby resulting in degraded performance due insufficient training data. This issue becomes even more challenging in the presence of label noise. This is because, in this scenario, it is hard to learn a good model from unreliable training data and select the desired model using unreliable validation data (Chen et al., 2021).
In this paper, we aim to pinpoint an appropriate early stopping point to mitigate the negative ef- fects of label noise without relying on any hold-out validation data. Our motivation stems from the observation that there exist qualitative differences between DNNs trained on clean examples and mislabeled examples (Arpit et al., 2017). By tracking these differences during the training process, we can pinpoint the moment when DNNs transition from primarily fitting clean examples to pri- marily fitting mislabeled examples, and thus determine an appropriate early stopping point. In what
Conresponding authors.

---

Published as a conference paper at ICLR 2024
5000
1000
(a)
Training Erros and Smoothed Prediction Changes
Test Error
3000
Training krao
25000
Training Error (clean examples)
Training (mislabeled examples)
20000
Predi prediction Changes
15000
(b)
10000
20000
epach learning
50
100 Epochs
125
150
175
200
Stage-1
Stage-2
Stage-3
Mothhaee Stmpphee
Qweetting
(d)
Stability+
Stability-
Stabilitt+
We examine how the model's fitting and generalization performance evolves during the training process of learning with noisy labels. Utilizing the k-epoch learning metric (Yuan et al., 2023), measure the number of training examples that be consistently classified according to their provided labels. This allows us to capture fluctuations in the model's fitting performance. We categorize the training process into three stages according to the stability of the fitting performance (panel d). This categorization is informed by an integrated analysis of generalization performance derived from test ervor (panel a) and fitting performance derived from training error (panel b) and 8-epoch learning metrics (panel c). Thus, we design the prediction changes metric to measure the shifts the model's predictions on the training set to pinpoint the early stopping point. For detailed information regarding the experiment settings, please refer to Appendix B and Section 3.
Figure 1:
follows, demonstrate in Figure 1 how to determine such a point by tracking the changes in the model's fattng performance, instead of leveraging any validation with hold-out data.
To accurately pinpoint appropriate early stopping point without relying hold-out data, introduce a metric called prediction changes (PC), as shown in Figure We use PC to measure the number examples whose predictions have changed between words, this metric quantifies the prediction stability degree of the model trained on a noisy training Therefore, this metric offers straightforward method for tracking the changes in the model's firring performance throughout the training process. These changes, metaphorically likened to wind waves, inspired the name of our method, "Label Wave".
successive epochs. In other
The effectiveness of the PC becomes evident when observing Stages and 2 in Figure I: the curve of PC in Figure ((b) exactly matches the curve of test error in Figure ((a). In Stage 1, as the model begins to fit clean examples (i.e., learns simple patterns), both the PC and the test error steadily decrease. In Stage 2, as depicted in training error in Figure 1(b), the model begins to fit mislabeled examples significantly, thereby leading to rise in test error. Concurrently, the model's fitting performance experiences notable fluctuations or even declines. Hence there an increase in the model's 's prediction fluctuations, leading to increase in PC. Thus, the local minimum of PC, marking the turning point between Stages and 2, is identified as the early stopping point.
In Stage 2, the rise in both PC and test error indicates that the model's efforts to minimize the total loss across the training set to accurately fit mislabeled examples not only undermine its ability to generalize to test data, but also harm its performance fit the correct patterns in the training data. In other words, fitting mislabeled examples impairs the overall model's fitting performance. This pronounced phenomenon Stage 2 is termed learning confusion patterns, reflecting the model's misinterpretation of the training data distribution.
2

---

Published as a conference paper at ICLR 2024
Notably, our proposed Label Wave method can identify the early stopping point at which the model transitions from primarily fitting clean examples to mislabeled ones. This can be achieved without prior knowledge about which training examples are mislabeled examples the need for separate hold-out validation data. This method not only provides a more efficient way to determine the early stopping point but also ensures sufficient training data to produce effective model.
Our main contributions can be summarized as follows:
1. We present an empirical observation that, when training DNNs from a noisy dataset, fitting mis- labeled examples impairs not only the generalization performance but also the overall model's s fitting performance.
2. Building on the correlation between the model's generalization and fitting performance estab- lished in our empirical analysis, we propose the Label Wave method, which uses the changes in the model's predictions on the training set to identify the early stopping point.
3. Through extensive experiments, we show both the effectiveness of the Label Wave method across various settings and its capability to enhance the performance of existing methods for learning with noisy labels.
RELATED WORK
Memorization & Forgetting. In deep neural networks, generalization is not solely dictated by the complexity of the hypothesis space (Chaudhari et al., 2019; Advani et al., 2020; Jiang et al., 2020). They can still generalize effectively even without regularizers, a characteristic termed as Memoriza- tion'. Building this, recent publications (Toneva et al., 2019; Feldman & Zhang, 2020) have studied the phenomenon known as the "forgetting event", aiming to understand how training data and network structures influence generalization. Based on the concept of forgetting, we observed that when the model starts to fit mislabeled examples, it begins to significantly forget the training data. This forgetting increases until it reaches turning point, after which the forgetting decreases.
Model Selection. Numerous indirect methods have been proposed for selecting an appropriate model. These methods include marginal likelihood estimator (Duvenaud et al., 2016), computed gradients (Mahsereci et al., 2017), leveraging unlabeled data (Garg et al., 2021; Deng & Zheng, 2021; Forouzesh et al., 2023), noise stability (Arora et al., 2018: Morcos et al., 2018; Zhang al., 2019), estimating generalization gap (Jiang et al., 2018; Corneanu et al., 2020), modeling loss dis- tribution (Song et al., 2019b; 2021; Lu & He, 2022), and training speed (Lyle et al., 2020; Ru al., 2021). In contrast to the existing methods, Label Wave focuses on the selection of an appropriate early stopping point during training process. Notably, this achieved without the need for additional or hold-out data and requires no preprocessing, ensuring low computational overhead.
Learning Stages. Traditional paradigms often posit the presence of two stages during the learning process: underfitting and overfitting (Goodfellow et al., 2016; Lin et al., 2023). Researchers explor- ing models trained on randomly labeled examples have divided the learning process into two distinct stages for a more nuanced perspective on deep learning: an initial stage of "learning simple patterns" (Arpit et al., 2017), followed by a subsequent stage of "memorization" (Zhang et al., 2017). Further findings, such epoch- epoch-wise wise Deep Double Descent (Belkin al., 2019; Nakkiran et al., 2021) and "Grokking" (Power et al., 2022; Liu et al., 2022b; Nanda et al., 2023), have highlighted the potential existence of additional learning stages under specific conditions. However, these studies primarily focus on classifying learning stages based on generalization performance when models learn from imperfect data, thereby overlooking variations in model fitting performance.
Learning Dynamics. A wealth of research endeavors have aimed to classify the "hardness" of examples by tracking learning dynamies, which is particularly beneficial for scenarios involving noisy labels or -ail distributions. Methods like Late Stopping (Yuan et al., 2023), SSLT (Maini et al.), Self-Filterig (Wei et al., 2022b), SELFIE (Song et al., 2019a), and ROCL. (Zhou al., 2021) exemplify this approach. However, these studies primarily focus on the dynamic changes of individual examples for the purpose of sample selection, rather than assessing the overall dynamics to distinguish different stages of the training process in learning with noisy labels.
"Memorization" is not formally defined, and we denote Memorization as training DNNs to fit the assigned lahel for each particular instance, in similar spirit Feldman & Zhang (2020) and Forouzesh al. (2023).
3

---

Published as a conference paper at ICLR 2024
3 DETERMINING EARLY STOPPING BY TRACKING FITTING PERFORMANCE
In this section, we aim to track the fitting performance to pinpoint the turning points model's generalization performance during the training process in the presence of noisy labels. To pro- vide a clearer view of the model's fitting performance, we incorporate two metrics: stability and variability metrics. With these metrics, we identify and describe the key turning points of fitting performance, and elaborate on their correlation with generalization performance. We further intro- duce transitional stage termed "learning confusing patterns" between the two turning points in fitting performance. Building on this foundation, we formally introduce our proposed Label Wave method, which determines an early stopping point by tracking the changes in the model's fitting performance. All experiments presented in this section use standard 18 backbone. For detailed information regarding the experiment settings, please refer to Appendix B.
Prior to an in-depth discussion, we first clarify three basic concepts (Goodfellow et al., 2016): (i) Test error, an empirical measure of "generalization performance". indicates a model's ability to make accurate predictions on previously unseen data. (ii) Training error, empirical measure of "fitting performance". indicates a model's 's ability to fit the training data. (iii) Training process:
Given model f(: that is being trained on a noisy training set D comprised of labeled examples (x,, yi) where i = 1, 1,2....,,, n, the model's predicted label for each training instance D; evolves as training progresses. After i epochs, this predicted label for w; is represented o/-
3.1 METRICS FOR TRACKING FITTING PERFORMANCE
In Figure 2, we track both the test error and the training error. From our analysis of the test error, we identify two pivotal points where there is a notable shift in generalization performance: Point 1, where the test error attains its global minimum, and Point 2, where the test error transitions to stable state after rising from this minimum. Notably, Point typically considered the optimal early stopping point. As determined by the test error, Point 1 indicates the model that generalizes best throughout the training process. Furthermore, an examination of the training error reveals that, after Point I, the model starts to significantly fit mislabeled training examples. This suggests that fitting mislabeled examples impairs the model's 's generalization performance.
Ret Brer Trahing Brews
Training (mielabeled exemples)
10000
Test Emro
Point 1
Point 2
5000
3000
100
130
200
Epochs
Figure 2: Tracking test error and training error (mislabeled examples) in training process.
In real-world applications, however, obtaining prior knowledge about which training examples are mislabeled examples is often impossible, thereby eliminating the possibility of relying on how well the model fits mislabeled examples climinate the dependency on hold-out datasets, implementing carly stopping based on validation errors becomes unfeasible. To address these challenges, focus on the following two metrics to evaluate the model's "fluctuations in predictions" on the training set.
criterion for early stopping. Furthermore,
we aim to
Stability Metric. We employ the k-epoch learning metric (Yuan et al., 2023) to quantify the stability of the model's predictions for specific subsets of examples, such as the clean data subset D.. A higher k-epoch learning value indicates that the model's predictions have greater stability in recent epochs and exhibit fewer fluctuations in predictions. Let acc; = of=y, be a binary variable that indicates whether the model accurately classifies example at epoch t. The stability of the model's predictions from epoch k + 1 epoch t can be quantified as follows:
k-epoch learning Σ iED:
ace/ A acc((-1) A A acc! acc((=k+2)
acc. acc (1-++1)
Variability Metric. In contrast to the stability metric, we use prediction changes as a variability metric to emphasize the model's inconsistency in classifying training set examples between the
4

---

Published as a conference paper at ICLR 2024
current epoch and the previous one. A lower value of prediction changes indicates greater stability and fewer fluctuations the model's predictions for the current epoch. The variability in the model's predictions from epoch 1 to epoch t can be quantified follows:
prediction changes
Σ
IED L (
(2)
3.2 FITTING MISLABELED EXAMPLES IMPAIRS THE MODEL'S FITTING PERFORMANCE
Figure 2 presents two approaches to selecting the best generalizing model using both test (validation) enror and training error on the mislabeled examples. However, these methods either require a separate hold-out set rely true labels in the training set. In contrast, method focuses on a unique concept: fluciuations in predictions (FIPs). By quantifying these fluctuations using the stability and variability metries, corresponding fitting performance change in two nearby points, which we defined in Section 3.1.
observe how the model's data-fitting strategy and its
25000
Predict Predict
Preskit changesto
17503
20020
15003
15000
Stabilitt
Varisblilty:
10003
Skabiliiy-
7500
Training accuracy (Slean ecamplpu
4-spoch leaming
2200
B-epoch leaming ing
Fluctuction examples
Paint
Poim
Point
Point
"aaing Eno (mtrabbled exampleu
Training Ener ie iiabbed examplest
10000
5xx10'
5'x
4x 10"
3x12
Erro
50
100
150
200
50
100
1:00
.0D0
Tpache
Epochs
(a) Stability metric, 8-epoch learning
(b) Variability metric, prediction changes
Figure 3: Using stability and variability metrics to track fluctuations in predictions.
In Figure 3, we chart these metrics throughout the training process of the model. Specifically, the stability metric is shown in Figure 3(a) and the variability metric Figure 3(b). Both metries are designed to track FIPs. Our observations highlight clear correlation between the metrics' turning points and the critical points determined by the shifts in generalization performance. This allows us to establish some key empirical findings:
Before Point 1, FIPs gradually decrease.
After Point FIPs shift from decreasing increasing.
After Point 2, FIPs shift from increasing to decreasing and eventually converge to steady state.
Our empirical observation reveals correlation between generalization performance and changes in the model's fitting strategy. We observe noticeable shifts in both performance and strategy before and after Point 1. After Point 1, the performance of models from subsequent epochs declines notice- ably in their ability to fit the examples from the training set. Moreover, the data from our 8-epoch learning indicates that more than half of the examples in D. (clean training examples) cannot be stably predicted after Point 1. This significant decline contradicts the assumption that fluctuations following Point I only stem from a small subset of challenging examples within D... Therefore, we term the phenomenon where the change in the model's 's fitting performance at the Point as fitting mislabeled examples impairs the overall model's 's fitting performance. This implies that by track- ing fluctuations in predictions, we can identify whether the model has started to overfit mislabeled training data just by observing changes in its fitting performance.
5

---

uncertainties Published as a conference paper at ICLR 2024
Interestingly, the model's behavior from Point 1 to Point 2 is noticeably different from its behavior before and after these points. The stage from Point 1 to Point 2 cannot simply be considered as either "incorporation of prior" (Bengio et al., 2009) "overfitting the dataset". Instead, to see this as a transitional stage in learning with noisy labels, situated between the widely accepted stages of "learning simple patterns" (Arpit et al., 2017) and "memorization" (Zhang et al., 2017). We have coined this transitional stage "learning confusing patterns". A more detailed discussion on implications of this stage and Point will be explored in Appendix A.
propose
3.3 LABEL WAVE METHOD
Based observations, we identified correlation between the model's fitting and generalization performance when learning with noisy labels. With this understanding, we formally introduce our proposed Label Wave method. This method utilizes the observed correlation between fluctuations in predictions and test error. We aim to pinpoint an appropriate early stopping moment by closely tracking the model's predictions on the noisy training set. This moment, referred as Point 1 in Section 3.1, is where the model begins to significantly fit mislabeled examples. To keep our method simple, intuitive, and computationally efficient, we directly refer the variability metric explored in section 3.1, which we term as prediction changes or PC.
Algorithm Flow. We use the prediction changes, represented as PC,, to track the changes in the model's fitting performance at the epoch f. The first turning point of PC, is identified as the early stopping point for model selection. practice, tracking PC, is often susceptible inherent in the training process. We enhance the robustness of our Label Wave method by utilizing Moving Averages techniques. By averaging PC, over the recent epochs (see Appendix for the sensitivity analysis of k), we derive more stable version, denoted as PC₁, as follows:
PC = (PC, + PC,-1 + PC2
As depicted in Algorithm 1, we compute PC, using the predictions of f(:; 0) on the noisy training set D. By applying the Moving Averages and Patience, identify the first local minimum point of the PC, and designate it as the early stopping point for the model selection, as follows:
Early Stopping Point
ffrst-min where firs-miin is the first local minimum of PC,
(4)
The Label Wave method allows pinpoint the moment when the model's fitting performance stability starts to decline by Based on our discussion in Section 3.2, stopping the training process such moment can prevent the model from beginning to overfit the mislabeled training data, ensuring the selection of a model with tooottal- generalization performance. This robust and intuitive method not only does not rely on validation sets and prior knowledge of training labels but also enhances the generalization of the selected model. To solidify our understanding and validate the efficacy of the Label Wave method, we embark a series of empirical experiments detailed in the upcoming section. For scenarios where the Label Wave method does not work, see the discussion in Appendix C.3.
in predictions and computing prediction changes.
Algorithm Label Wave
Let e, be the initial parameters and v be the local minimum of PC. Let be the "Patience", repre- senting the number of times worsening PC is observed before halting.
0.. 1 + 0, i <- 0, v <- 00
1: while i < p do
2: Update 0 by running the training for n steps, and <- + n.
3: PC, +- Compute prediction changes (PC) in step l.
4:
PC; <- Moving Averages s PC recent & steps. then
5: PC,
6: U + piiit O, 0, t-
// Models stored at every new local minimum.
7: else
8:
i +-1+1 // Counting Patience when PC, is larger than local minimum.
9:
end if
10: end while=0
Best parameters are 0" and best number of training steps is t..

---

Published as a conference paper at ICLR 2024
4 EXPERIMENTS
In this section, we empirically demonstrate the effectiveness of our Label Wave method. In Section 4.1, we conduct comprehensive experiments to validate the broad applicability of our method across different settings. Section 4.2 highlights the capability of the Label Wave method to enhance the generalization performance of existing learning with noisy labels methods. Furthermore, in Section 4.3, we examine the Kendall correlation between our method and test accuracy. This analysis is compared with empirical validation on hold-out data. The findings suggest that Label Wave amplifies the generalization performance of learning with noisy labels methods not only by leveraging more training data but also by improving the precision in identifying early stopping points.
4.1 EFFECTIVENESS OF LABEL WAVE
Building upon the Label Wave method experiments conducted using the ResNet- 18 backbone, we performed additional tests. These experiments demonstrate the consistent efficacy of our proposed Label Wave method across a wide range of settings, including multiple datasets, diverse network ar- chitectures, a range of parameters, various optimizers, and different levels and types of label noise. As evident in Table 1, there is only a slight difference between the test accuracy of models selected by the Label Wave method and the global maximum test accuracy during the training process. This indicates that the Label Wave method consistently selects models, irrespective different compo- nents and parameters, that are at or near the global maximum in test accuracy. More experimental results and details of experiment settings can found Appendix C.
Noise. We assessed the capability of Label Wave method to handle learning from various levels and types of label noise. These include 10% to 80% of the incorrect labels with the Instance- dependent noise (abbreviated Ins.) (Xia et al., 2020) and Symmetric noise (abbreviated (Van Rooyen et al., 2015). Notably, with CIPAR-N (using its "Random 1/2/3" and "worst" setting in CIFAR-ION and "worst" setting in CIFAR-IOON, characterized by approximately 40% real-world noise) (Wei et al., 2021), we evaluate effectiveness of Label Wave method on real-world noise.
Sym.)
Architectures. We verified the effectiveness of our method in learning with noisy labels sev- eral commonly used deep learning models, including ResNer (He et al., 2016), VGG (Simonyan & Zisserman, 2014), Inception-v3 (Szegedy al., 2016), and DenseNet (Huang et al., 2017).
Datasets. Our evaluations also confirmed the effectiveness of Label Wave method across multi- datasets. These datasets comprise seven vision-oriented sets: CITAR-1O. CIFAR-100 (Krizhevsky et al., 2009), CIFAR-N (Wei et al., 2021), Clothing / M (Xiao et al., 2015), WebVision (Li et al., 2017). Food101 (Bossard et al., 2014), and Tiy-ImageNer (Le & Yang, 2015), along with a rext-oriented dataset: NEWS (Kiryo et al., 2017; Yu et al., 2019). We tested our method on class imbalanced datasets CID-CE and with using class imbalanced method CID-LDDAM (Cao al., 2019).
Parameters & Optimizers. The robustness of the Label Wave method is validated by testing it with different learning rates, batch sizes, random seeds, and optimizers. By adjusting the batch sizes to 64, 128, 256, learning rates to 0.01, 0.005, 0.001, random seeds to 1, 2, 3, 4, 5, and employing different optimizers such as SGD with momentum (Robbins & Monro, 1951; Polyak, 1964), RM- Sprop (Tieleman et al., 2012), and Adam (Kingma & Ba, 2014), demonstrate that the Label Wave method remains resilient to reasonable variations in both parameters and optimizers.
Table I: Differences (meantstd) among the model selection methods. Lower is better.
Noise (Sym.)
20%
30%
40%
50%
60%
70%
80%
Global Maximum (%)
84.08++0.10 01.73+0..3
80.45-00.6 75.48 1..488-0444 0.46
08.5 57.57++0.98 37 0.92
Label Wave (%)
83.43:1..32 $1.48.10.36 10.15
75.20
68.21 56.6411.57 37. 37.98
Difference
0.65% %
0.25% %
0.30%
0.28%
0.36% %
0.93%
0.38%
Noise (Ins.)
20%
30%
40%
50%
60%
70%
80%
Global Maximum
84.28-10.45 82.83-+:0.65 77. 0..6
6..74=0.78
1.52 29.48-
20.59:+0.69
Label Wave (%)
83.99-00.72 82.68-+0.44 76.81 63.87-0068
45.92++2.00 28.87:40.69 1.06
Difference
0.30% %
0.15 %
0.88%
0.87%
0.88 %
0.61 %
0.50%

---

Published as a conference paper at ICLR 2024
Noise& & Datasets
Random
Rondom 2
Random 3
Ins. 10%
Sym. 10%
CID-CE
CID-LDAM
Global Maximum (%)
83,21 82.86.10.24 03.25 85.91
85,4610.12
56.03
61.8010993
Label Wave (%)
83.03=0.35 82.53:0.223
82.92:0..50
0..56
84.92:0.06
55.31
61.07:0.88
Difference
0.18%
0.13%
0.18%
0.35%
0.54%
0.72%
0.73%
Datasets
CIFAR-IO CIFAR-100 CIFAR-ION CIFAR-IOON
NEWS
Imay-ee
Adam
Global Maximum (%)
79.99-0.57 49.92+0.81 76.23+0.17 47.48+0.49
40.58
34.88+0.15
79.12+0.55
Label Wave (%)
79.54-0.91 49.39+0.69 75.98+0.49 47.04+0.38 4..06+1.15 34.20+0.42
78.92+0.66
Difference
0.45%
0.52%
0.25%
0.45%
0.51%
0.68%
0.21 %
Architectures
ResNet-18 ResNet-34
Re
ResNet-101
VGG-16
Inception-v3
Dense-
Global Maximum (%)
79.86=0.52 +0
19.4
77.35+0.40
78.00=00.53
57.23++0.52
66.09++0222
Label Wave (%)
79.36=0.77 0.
78.94
06.9910.75
77.77-1043
56.80.10.73
65.8 65.87110.88
Difference
0.50%
0.31%
0.49%
0.36%
0.23%
0.43%
0.22%
Parameters
LR. 0.01
LR. 0.005
LR 0.001
BS. 64
BS. 128
BS. 256
RMSprop
Global Maximum (%) Label Wave (%)
80.08 0.779
72.55
80.2310.57
80.11
76.68:1066
76.7910.61
80.38=0557 76.67=0.95 72. 0.555
80.59+0441
79.69=00.68
76.244+0.49
0..22++0.95
Difference
0.30%
0.31%
0.40%
0.36%
0.43%
0.64%
0.57%
4.2 ENNIANCING EXISTING LEARNING WITH NOISY LABEL METHONS
To verify the effectiveness of our method in practical applications, we apply our proposed Label Wave method within range of state-of-the-art learning with noisy labels methods. Among the methods we have explored are robust loss functions (Feng et al., 2020), robust regularization (Xia et al., 2021; Wei et al., 2022a), label noise correction (Liu et al., 2020; Cheng et al., 2021), 1), sparsity over-parameterizztion (Liu et al., 2022a), and the baseline Cross-Entrop (Rubinstein, 1999).
As shown in Tables and 3, we conducted evaluations using ResNet-18 on CIFAR-IO and ResNet- 34 on CIFAR-100 (Krizhevsky et al., 2009; He et al., 2016), both tainted with 40% Symmetric noise (abbreviated as Sym.) (Van Rooyen et al., 2015). Our primary assessment metric was the test accuracy of the selected model, determined by the average and standard deviation over five runs. Notably, the Label Wave method consistently outperformed conventional validation methods employing 5% to 30% hold-out data. This indicates that the Label Wave method works as a more effective early stopping method than conventional hold-out data validation, thus augmenting the generalization performance of models trained by existing learning with noisy label methods. Results under more experimental settings can be found in the appendix D.
Table 2: Test accuracy (mean std) of each method on CIFAR-10 (with 40% Sym. label noise).
Method
Val. 5%
Val. 10%
Val. 20%
Val. 30%
Label Wave
CE (Rubinstein, 1999)
80.29:10.66%
79.93+1..22%
78.94:0.52%
7..3%
81.61 %
Taylor-CE (Feng et al., 2020)
83.85++0.619 %
83.28:10.34%
82.33+10.43%
12%
85.06=10.00 %
ELR (Liu al., 2020)
90.23++0.7%%
88.99:00.41%
88.43+t0.38%
87.24+:0.415
90.45:+..52%
CDR (Xia et al., 2021)
86.45-10.36%
85.98110.43%
85.60 500%
83.71
87.69-0.100%
CORES (Cheng et al., 2021)
0..2%%
86.9=0.25%
85.78++0.69%%
85.17-0.55%
0.17%
NLS (Wei et al., 2022a)
0..62%
81.89-00.18%
81.12-00.53%
0..23%
13.49%
SOP (Liu et al., 2022a)
86.40
86.8010.77%
87.12
86.37-0.66%
88.42.10.38%
Table 3: Test accuracy (tean=ist)) of each method on CIFAR-100 (with 40% Sym. label noise).
Method
Val. 5%
Val. 10%
Val. 20%
Val. 30%
Label Wave
CE (Rubinstein, 1999)
49.69+0.89%
48.42+00.00%
45.96+1.17%
43.65+0..33%
50.96+0.300 %
Taylor-CE (Feng et al., 2020)
56.67+0.35%
56.48+0..188
55.23+00.50%
54.38+0.2995
57.64+ 0.28
ELR (Liu et al., 2020)
64.53:10.36%
62.05110.0%%
60.09:10.74%
57.49-0.42%
65.36:10.39 %
CDR (Xia et al., 2021)
61.93+0.38%
60.82++0.88%
58.27+
54.90+
63.34+0..55%
CORES (Cheng et al., 2021)
44.32-10.82%
44.48:0.35%%
43.32-10.57%
41.28 0.4%%
45.03:10.38 %
NLS (Wei et al., 2022a)
57.08
55.89410.269
54.09:10.5%%
52.82:00.28%
58.05:0.155%
SOP (Liu et al., 2022a)
67.27%0.45%
6..28++0611%
64.46-40.48%
63.02-+0.48%
68.53+00.30%
8

---

Published as a conference paper at ICLR 2024
4.3 CORRELATION WITH TEST ACCURACY
In this subsection, we provide evidence that our proposed method exhibits superior performance compared hold-out validation across a diverse range of validation sizes and noise rates. We employ the same experimental settings as in Section 3.
Hold-out Set Sizes. We utilized the same subset of training data to calculate both prediction changes (PC) and hold-out validation. This implies that we only use limited-size subset from the training data to compute PC. the lower subfigure of Figure 4(a), the Kendall correlation (Kendall, 1938) is illustrated, showcasing the association between the ranking of model selection and actual test accuracy for both methods. the upper subfigure of Figure 4(a), we focus on the test accuracy of models selected by both the Label Wave and hold-out validation methods. Our experiments emphasize that increasing the size the hold-out set improves precision model selection. However, the size of the hold-out set increases, the data available for training decreases, leading to a reduction in test accuracy for candidate models obtained during the training process. These findings suggest that our proposed Label Wave method can select models more precisely and yield better test accuracy, even when computed on smaller set.
Noise Rates. We further compare the test accuracy of models selected by our proposed Label Wave method and the test accuracy of models selected by hold-out validation in various set sizes with varying noise rates. As shown in Figure 4(b), demonstrate the gain in model test accuracy selected by the Label Wave method. This underscores the robustness and superior efficacy of the Label Wave method, especially in high levels of noise and large set sizes scenarios.
Getund Turhh Arc.
Aug 10. (80
Aug Nostal Cone abon
6.0
0.6
0.4
Validation
Wave
25
80
160
257
570
1006
2002
4230
X200
10002
Siet Stoe/1oo
(a) Test accuracy and of compared methods
(b) Test accuracy gain of Label Wave
Figure 4: We aim to compare the test accuracy of models selected at the early stopping point by the Label Wave method and those selected by the hold-out validation method. (a) We selected a subset of the training data, with set sizes ranging from 250 to 16,000. This subset was used for both to compute the prediction change in the Label Wave method and serving as the hold-out set hold-out validatioo, respectively. We computed the Kendall correlation and the test accuracy of the models selected by these two methods. (b) Further analysis was conducted to evaluate the difference in test accuracy between models selected by our proposed Label Wave method and those selected by the hold-out validation method with noise rates ranging from 20% to 60%.
CONCLUSION
In this paper, we introduced the Label Wave method for early stopping in the presence of label noise. Our proposed method uses the prediction changes metric to track changes in a model's fitting performance pinpoint the early stopping point, eliminating the need for separate hold-out data. Extensive experiments showcased the effectiveness of the Label Wave method across various settings, leading to improved generalization performance in methods for learning with noisy labels. Furthermore, we introduced transitional stage in learning with noisy labels, denoted as learning confusing patterns. Looking forward, there is potential to devise an advanced early stopping metric that could surpass the capabilities of the prediction changes metric, thereby enhancing the Label Wave method. A deeper dive into the transitional stage may offer pivotal insights into how deep neural networks behave when learning with noisy labels.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
The authors thank the anonymous reviewers and area chairs for their insightful and helpful com- ments. Suqin Yuan especially thanks Muyang Li, Runqi Lin, and Yingbin Bai who gave helpful ad- vice during the rebuttal. Tongliang Liu is partially supported by the following Australian Research Council projects: FT220100318, DP220102121, LP220100527, LP220200949, and IC190100031.
REFERENCES
Madhu S Advani, Andrew M Saxe, and Haim Sompolinsky. High-dimensional dynamics of gener- alization error in neural networks. Neural Nenworks, 2020.
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. In ICML, 2018.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In ICML, 2017.
Yingbin Bai, Erkun Yang, Bo Han, Yanhua Yang, Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Understanding and improving early stopping for learning with noisy labels. NeurIPS, 2021.
Peter L. Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 2020.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical biai-variance trade-off. Proceedings of the National Academy of Sciences, 2019.
Yoshua Bengio et al. Learning deep architectures for ai. Foundations and trends@ in Machine Learning. 2009.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative com- ponents with random forests. In ECCV, 2014.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution--waaee margin loss. NeurlPS, 2019.
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019.
Pengfei Chen, Junjie Ye, Guangyong Chen, Jingwei Zhao, and Pheng-Ann Heng. Robustness of accuracy metric and its inspirations in learning with noisy labels. In AAAI, 2021.
Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance- dependent label noise: A sample sieve approach. In ICLR, 2021.
Ciprian A Corneanu, Sergio Escalera, and Aleix M Martinez. Computing the testing error without testing set. In CVPR, 2020.
Weijian Deng and Liang Zheng. Are labels always necessary for classifier accuracy evaluation? In CVPR, 2021.
David Duvenaud, Dougal Maclaurin, and Ryan Adams, Early stopping as nonparametrie variational inference. In Proceedings of the 19th International Conference Artificial Intelligence and Statistics, 2016.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. NeurlPS, 2020.
Lei Feng. Senlin Shu, Zhuoyi Lin, Fengmao Lv, Li Li, and Bo An. Can cross entropy loss be robust to label noise? In I/CAI, 2020.
10

---

Published as a conference paper at ICLR 2024
Mahsa Forouzesh, Hanie Sedghi, and Patrick Thiran. Leveraging unlabeled data to track memoriza- tion. In ICLR,, 2023.
Saurabh Garg, Sivaraman Balakrishnan, Zico Kolter, and Zachary Lipton. Ratt: Leveraging unla- beled data to guarantee generalization. In ICML, 2021.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Bo Han, Quanming Yao, Tongliang Liu, Gang Niu, Ivor W Tsang, James T Kwok, and Masashi Sugiyama. A survey of label-noise representation learning: Past, present and future. arXiv preprint arXiv:201 1.04406, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinherger. Densely connected convolutional networks. In CVPR, 2017.
Zhuo Huang, Chang Liu, Yinpeng Dong, Hang Su, Shibao Zheng, and Tongliang Liu. Machine vision therapy: Multimodal large language models can enhance visual robustness via denoising in-context learning. arXiv preprint arXiv:2312.02546, 2023.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In /CML, 2015.
Yiding Jiang. Dilip Krishnan, Hossein Mobahi, and Samy Bengio. Predicting the generalization gap in deep networks with margin distributions. arXiv preprint 1rXiv:/810.00113, 2018.
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. In ICLR, 2020.
Maurice G Kendall. A new measure of rank correlation. Biometrika, 1938.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 141.v:1412.69880 2014.
Ryuichi Kiryo, Gang Niu, Marthinus Du Plessis, and Masashi Sugiyama. Positive-unlabeled learning with non-negative risk estimator. NeurlPS, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 2015.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 2015.
Wen Li, Limin Wang, Wei Li, Birikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint
2017.
Runqi Lin, Chaojian Yu, Bo Han, and Tongliang Liu. On the over-memorization during natural, robust and catastrophic overfitting. arXiv preprint arXiv:2310.08847, 2023.
Sheng Liu, Jonathan Niles- Weed, Narges Razavian, and Carlos Gemanddz-Grandde Early-lcariing regularization prevents memorization of noisy labels. NewrIPS, 2020.
Sheng Liu, Zhihui Zhu, Qing Qu, and Chong You. Robust training under label noise by over- parameterization. In JCML, pp. 14153-14172, 2022a.
Ziming Liu, Ouail Kitouni, Niklas S Nolte, Eric Michaud, Max Tegmark, and Mike Williams. To- wards understanding grokking: An effective theory of representation learning. NeurlPS, 2022b.
Yangdi Lu and Wenbo He. Selc: self-ensemble label correction improves learning with noisy labels. arXiv preprint ar arXiv:2205.01 156, 2022.
Clare Lyle, Lisa Schut, Robin Ru, Yarin Gal, and Mark van der Wilk. A bayesian perspective on training speed and model selection. NeurlPS, 2020.

---

Published as a conference paper at ICLR 2024
Maren Mahsereci, Lukas Balles, Christoph Lassner, and Philipp Hennig. Early stopping without validation set. arXiv preprint arXiv:/703.09580, 2017.
Pratyush Maini, Saurabh Garg, Zachary Chase Lipton, and Zico Kolter. Characterizing datapoints via second-split forgetting. In JCML 2022: Workshop on Spurious Correlations, Invariance and Stability.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018.
Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. NeurIPS, 2014.
Ari Morcos, David GT Barrett, Neil C Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. arXiv preprint axXiv:103.06690 2018.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021.
Neel Nanda, Lawrence Chan, Tom Liberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXhv:2301.05217, 2023.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, 2014.
Boris T Polyak. Some methods of speeding up the convergence of iteration methods. Ussr compu- tational mathematics and mathematical physics, 1964.
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- eralization beyond overfitting on small algorithmic datasets. arXiv preprint arX1v:2201.02177. 2022.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathemati- cal statistics, 1951.
Robin Ru, Clare Lyle, Lisa Schut, Miroslav Fil, Mark van der Wilk, and Yarin Gal. Speedy perfor- mance estimation for neural architecture search. NeurIPS, 2021.
Reuven Rubinstein. The cross-entropy method for combinatorial and continuous optimization. Methodology and computing in applied probability, 1999.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:14099 1556, 2014.
Hwanjun Song, Minseok Kim, and Jae-Gil Lee. Selfie: Refurbishing unclean samples for robust deep learning. /CML, 2019a
Hwanjun Song, Minseok Kim, Dongmin Park, and Jae-Gil Lee. How does early stopping help generalization against label noise? arXiv preprint arXiv:1911.0805) 2019b.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Robust learning by self-transition for handling noisy labels. In KDD, 2021.
Hwanjun Song, Minscok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEEE Transactions on Neural Networks and Learning Systems, 2022.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 2014.
Christian Szegedy. Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
12

---

Published as a conference paper at ICLR 2024
Tijmen Tieleman, Geoffrey Hinton, et al. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, 2012.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. An empirical study of example forgetting during deep neural network learning. In /CLR, 2019.
Brendan Van Rooyen, Aditya Menon, and Robert C Williamson. Learning with symmetric label noise: The importance of being unhinged. NeurIPS, 2015.
Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learn- ing with noisy labels revisited: A study using real-world human annotations. arXiv preprint 2r 10. 12088, 2021.
Jiaheng Wei, Hangyu Liu, Tongliang Liu, Gang Niu, Masashi Sugiyama, and Yang Liu. To smooth or not? when label smoothing meets noisy labels. In ICML, 2022a.
Qi Wei, Haoliang Sun, Xiankai Lu, and Yilong Yin. Self-filterig: noise-aware sample selection for label noise with confidence penalization. In ECCV, 2022b.
Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? NeurIPS, 2019.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang. Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. NeurIPS, 2020.
Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong, Nannan Wang, Zongyuan Ge, and Yi Chang. Robust early-learning: Hindering the memorization of noisy labels. In ICLR, 2021.
Xiaobo Xia, Pengqian Lu, Chen Gong, Bo Han, Jun Yu, and Tongliang Liu. Regularly truncated m-estimators for learning with noisy labels. IEEE Transactions Puttern Analysis and Machine Intelligence, 2023.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In CVPR. 2015.
Yun Xu and Royston Goodacre. On splitting training and validation set: a comparative study of cross-validation, bootstrap and systematic sampling for estimating the generalization performance of supervised learning. Journal of analysis and testing. 2018
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In ICML, 2019.
Suqin Yuan, Lei Feng, and Tongliang Liu. Late stopping: Avoiding confidently learning from mislabeled examples. In ICCV, 2023.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. ICLR, 2017.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
Jie Zhang, Earl T Barr, Benjamin Guedj, Mark Harman, and John Shawe-Taylor. Perturbed model validation: A new framework to validate model relevance. 2019.
Tianyi Zhou, Shengjie Wang. and Jeff Bilmes. Robust curriculum learning: from clean label detec- tion to noisy label self-correction. ICLR, 2021.
13

---

Published as a conference paper at ICLR 2024
A DISCUSSION ON learning confusing patterns
In this section, we delve into Point 2, where the model's test error transitions from increasing to sta- bilizing, by utilizing the metrics introduced in Section 3. Moreover, we introduce a novel transitional stage in learning with noisy labels, termed learning confusing patterns.
TE00
Steges Truining
0043
Loarn Simpla Patters
Learn Simple Patterss
Teary Emrt Srd Ledd W0
30009
Tuining Accuriy
Garly Stouping
Accurayy irilate
Letdl Mave Drse
Loars Contasing Pattoras
Memorization
20003
Memorization
Memorization
12500
D000 can meeeoribe rasduntty Jubeire data Thang
DWVS do not jusr montoriiz rsal dlisa
EXNU do not just mevenriee randonnly ludeledd dala
(a) Tracking model's behavior between two points
(b) Our proposed new transitional stage
Figure 5: Based on the multi-metrics we are tracking for the model's generalization and fitting performance between Point and Point 2 (as shown in panel a), we propose a new transitional stage of learning with noisy labels, termed "learning confusing patterns" (shown in panel b).
As shown in Figure 5(a), when tracking fluctuations in predictions using stability (8-epoch learn- ing) and variability (prediction changes) metrics, we noticed Point is a turning point in the model's fitting performance, similar to the early stopping point, Point 1. Specifically, in Point 2, the fluc- tuations in predictions transitioned from increasing to decreasing trends, eventually stabilizing at a specific value. Here, empirically explain the evolution of fluctuations in predictions during the training process. Before reaching Point 1, as training progressed, the model's generalization perfor- mance improved. The reduction in fluctuations in predictions suggests that model with superior generalization is more noise-resistant (Arora et al., 2018; Morcos et al., 2018; Zhang et al., 2019; Forouzesh et al., 2023), resulting more stable fits. After Point 2, the model began its shift towards memorization. This decrease in fluctuations in predictions reflects the beginning of the model fitting to the assigned label for each specific example (Arpit et al., 2017; Toneva al., 2019; Feldman & Zhang, 2020; Xia et al., 2023), showing more consistent fits.
Nevertheless, the increase in fluctuations in predictions between Point and Point suggests the model is neither purely overfitting nor excellent in generalization during this stage. As shown in Figure 5(b), we deseribe this stage as the "learning confusing patterns", during which the stability of the model's predictions persistently decrease. Between these two points, the model's effort to minimize losses across the noisy training set causes it to adapt to mislabeled examples. This inte- gration of incorrect information into generalization compromises the model's 's overall representation of the noisy training set, leading to an increase in fluctuations in predictions and a concurrent decline in both generalization and fitting performance. In contrasting the earlier stage before Point I where the model "learn from simple patterns", the learning paradigm during learning confusing patterns can be termed "learn from misled patterns", indicating that, during this stage, while the model still generalizes by learning patterns from the data, the patterns it learns are overly complex and incorrect due to the model try generalize to mislabeled samples.
Building upon this, we identify Point 2 as the critical point where either generalization or memo- rization takes precedence. Beyond Point 2, the fluctuations in predictions begin notably decline, indicating the model's shift towards fitting individual example labels and achieving stable fitting across the entire noisy training set. However, the marked transition at Point 2, from misguided generalization to overfitting, cannot be satisfactorily explained by solely relying on our predefined stability and variability metrics. This presents an intriguing open question for further investigation.
14

---

Published as a conference paper at ICLR 2024
B DETAILED OVERVIEW OF BASIC MODEL IN OUR EXPERIMENTS
Architecture
Framework:
PyTorch, Version 1.11.0.
Model Type: Standard ResNet-18, sourced from the PyTorch torchvision library.
Dropout: Aligned with the standard ResNet-18, we do not incorporate dropout.
Parameters
Seed:
Batch Size: 128
Learning Rate: Fixed at 0.01.
Optimizer: Employs opt im. SGD with momentum 0.9.
Loss Function: Utilizes the ropssoossopyL.os from the module.
Dataset & Pre-processing
Dataset:
The CIFAR-IO dataset, which accessible via the torchvision datasets module. 20% of the training data is held out for validation during the training process.
Normalization: Leveraging the torchvision, transforms module, we normalize all pixel values fit within the [0.1] range.
Cropping: Images undergo random cropping procedures. For any 32x32 image, potential padding might be added, from which random 32x32 crops are then extracted.
Rotation: Images are subjected to random rotations, limited to a range of +15 degrees. This step ensures the model's robustness against various image orientations.
Label Noise
Symmetric Noise: We infused 40% symmetric noise into 80% of the CIFAR-10 training set, affecting 40,000 examples.
15

---

Published as a conference paper at ICLR 2024
C EFFECTIVENESS OF LABEL WAVE
In this appendix, we delve into the comprehensive details of experiments validating the effective- ness the Label Wave method within various noisy label learning scenarios. We have presented specific results from each setting in the main content (see Table in Section 4.1). However, here, we will discuss in-depth the experimental setups. Our experiments spanned a wide array of settings commonly employed in existing methods for learning with noisy labels. This encompassed varied architectures, datasets, noise types and levels, parameters, and optimizers. The overarching aim of these diverse setups ronments, Unless otherwise noted, the parameters and components remain consistent with the basic model in Appendix B outside of the each declared setting.
gauge the effectiveness of the Label Wave method across different envi-
C.I NETWORK ARCHITECTURES
In our exploration, we employed a selection of prominent deep learning architectures to validate the versatility and effectiveness of the Label Wave method:
ResNet (He et al., 2016): Renowned for its profound success in deep learning endeavors, especially in the realm of image classification, we engaged with multiple variants based on the original paper: ResNet-18, 18, ResNet-34, ResNer-50, and ResNer-I01.
VGG (Simonyan & Zisserman, 2014): Known for its straightforward and effective archi- tecture, we employed the VGG-16. The architecture's clarity and consistency enabled to assess the Label Wave method's s effectiveness with well-established feature hierarchy.
Inception-v3 (Szegedy al., 2016): With its unique design, Inception-v3 offers diverse receptive fields without taxing computational resources. Using this architecture, we evalu- ated our Label Wave method in networks optimized for multi-scale feature extraction.
DenseNet (Huang al., 2017): By employing the densely connected convolutional archi- tecture of DenseNet-122 121, we aimed to assess how the Label Wave method performs in network characterized by robust gradient flow due its dense connections.
C.2 DATASETS EMPLOYED
To assess the Label Wave method's versatility, we employed several datasets, widely recognized in the learning with noisy labels domain:
Vision-oriented Datasets:
CIFAR-1O and CIFAR-100 (Krizhevsky et al., 2009): Established benchmarks for im- age classification; CIFAR-10 contains 10 classes and CIFAR-I00 contains 100 classes, respectively, and have 50,000 training examples and 10,000 test examples.
CIFAR-N (Wei et al., 2021): An augmented version of the CIFAR dataset, "worst" setting, which is marked by approximately 40% real-world label noise, and presented an opportunity test the resilience of the Label Wave method.
it
Tiny-ImageNe (Le & Yang, 2015): With 200 classes, each class contains 500 training images, making a comprehensive dataset for evaluating performance.
Text-oriented Dataset:
NEWS (Kiryo et al., 2017; Yu et al., 2019): We leveraged the NEWS datasct to ex- amine the Label Wave method's applicability beyond images. The dataset consists of news articles categorized into various themes. For NEWS, we borrowed the pre- trained word embeddings from Pennington et al. (2014), and 3-layer MLP is used with Softsign active function.
Real-world Scenarios Dataset:
CIFAR-N (Wei et al., 2021): An augmented version of the CIFAR dataset, it "Random 1, 2, 3" setting, which is marked by approximately 20% real-world label noise, and presented an opportunity test the resilience of the Label Wave method.
16

---

Published as a conference paper at ICLR 2024
Clothing (Xiao et al., 2015): A large-scale dataset that focuses clothing clas- sification. It contains over million images of clothing items, categorized into 14 classes.
Web (Li al., 2017): Designed mirror the real-world challenges web- based image recognition, this dataset contains images collected from the internet. It comprises 2.4 million images from 1,000 different classes, mimicking the class distri- bution of ImageNet.
Food10/ (Bossard et al., 2014): This dataset is focused food recognition, consisting of 101 food categories, with 101,000 images. Each category contains 750 training images and 250 test images. The images are not artificially labelled and thus contain some level of real-world noise.
We tested our method on class imbalanced datasets, by setting the Imbalance Factor to 0.1 and the Noise Ratio (Sym.) to 0.4. Our experiments were conducted using our method in Cross Entsopy (Rubinstein, 1999) (CID-CE) and class im- balanced method LDAM (Cao et al., 2019) (CID-IDAMM).
Class imbalanced dataset:
Table 4: Differences (meanestd) among the model selection methods. Lower is better.
Datasets
Clothing IM
WehVision
Food101
Global Maximum (%)
70.56
57.58:10.14 57.24
80.73-1:.446
Label Wave (%)
70.12:0.34
80.12:1.01
Difference
0.44%
0.34%
0.61 %
C.3
LABEL NOISE
We examined the Label Wave method's effectiveness across various label noise conditions:
Symmetric Noise (Sym.) (Van Rooyen et al., 2015): Symmetric noise levels were var- ied between 20% to 80%, which provided environments to examine how the Label Wave method works with data that has uniformly distributed noise across different labels.
Instance-dependent Noise (Ins.) (Xia et al., 2020): By introducing 20% to 80% instance- dependent noise, we aimed examine how Label Wave method works with data that has noise correlated with the instance characteristics.
Real-world Noise (Wei et al., 2021): We have detailed the CIFAR-N dataset that is char- acterized by real-world noise. Building on that foundation, we have evaluated how Label Wave method performs when confronted with noise in real-world scenarios.
Low noise rate: We augmented the CIFAR-10 dataset with 10% symmetric noise (Sym. 10%) and 10% instance-dependent noise (Ins. 10%), presenting an opportunity to test the resilience of the Label Wave method.
Label Wave method not applicable very low or no label noise. There are many situations where validation and test errors consistently decrease even with (low level) noisy labels in the train- ing data. Modern deep neural networks often exhibit benign overfitting (Bartlett et al., 2020), a phenomenon also describable
memorization effect (Zhang et al., 2017; Arpit et al., 2017).
The effectiveness of the Label Wave method identifying an appropriate early stopping point is attributed to design of practical metric that tracks the significant onset of learning confusion patterns, namely prediction changes (PC). Therefore, the training process lacks a stage of learn- ing confusion patterns, such when training with perfect data employing robust regularization approaches, the original Label Wave method may not identify an appropriate stopping point.
However, it is important note that in these scenarios, applying early stopping to improve the model's 's generalization performance might not be necessary.
17

---

Published as a conference paper at ICLR 2024
C.4 PARAMETERS AND OPTIMIZATION
Our experiments encompassed an array of parameters and optimizers:
Batch Sizes (BS.): 64, 128, 256.
Learning Rates (LR.): 0.01, 0.05, 0.001.
Optimizers:
SGD with momentum (Robbins & Monro, 1951; Polyak, 1964); widely-used opti- mizer, we gauged how its momentum-based optimization worked in tandem with the Label Wave method.
RMSProp (Tieleman et al., 2012): Known for its robustness and flexibility in non- stationary settings, we examined its synergy with the Label Wave method.
Adam (Kingma & Ba, 2014): Recognized for its adaptability in adjusting learning rates during training, we determine its synergy with the Label Wave method.
Seeds:
Each experiment is rum five times, with the seed set to: 1, 2, 3, 4, 5, respectively.
C.5
REGULARIZATION TECHNIQUES
We examined the Label Wave method's effectiveness across various regularization techniques:
Mixup (Zhang et al., 2018): Mixup is an innovative data augmentation technique that operates by creating virtual training examples. It linearly interpolates between pairs of examples and their labels, effectively encouraging the model to favor simple linear behavior in-between training examples.
Batch Normalization (BN) (Ioffe & Szegedy, 2015): Batch Normalization is a widely adopted technique in deep learning, designed stabilize and accelerate the training pro- cess. By normalizing the inputs of each layer across the mini-batch, it addresses the issue internal covariate shift.
Dropout (Srivastava al., 2014): Dropout is regularization technique that prevents over- fitting by randomly deactivating subset of neurons during training. By doing this, it effectively creates network of varying architecture each time batch is passed through, which discourages the network from relying too much any single neuron and promotes better feature learning.
Data Augmentation (DA):
Cropping:
Images undergo random cropping procedures. For any 32x32 image, po- tential padding might be added, from which random 32x32 crops are then extracted.
Images are subjected to random rotations, limited to range of +15 degrees. This step ensures the model's robustness against various image orientations.
Rotation:
Table 5: Test accuracy (mean ittd) of each techniques on CIFAR-10 (with 40% Sym. label noise).
Regularization Techniques
Val. 20%
Label Wave
Global Maximum
Label Wave
63.00++0.66%
66.79-00.3%%
67.15++0.49%
Label Wave + BN + DA
78.94
81.61
81.76:10.30%
Label Wave + BN + DA Dropout
81.30++1.07%
83.57=0.24%
83.77=1.32%%
Label Wave + BN + DA + Mixup
81 81.06++0.499
82.38
83.09++.222%
Label Wave + BN + DA + Dropout + Mixup
81 81.66:L0.07%
83.67=0.45%
84.05-10.35%
18

---

Published as a conference paper at ICLR 2024
C.6 QUANTITATIVE COMPARISON BETWEEN LABEL WAVE AND GLOBAL MAXIMUM TEST ACCURACY
In Table I, we use "Difference" to indicate the disparity in test accuracy between the model that achieved the Global Maximum test accuracy throughout the training process and the model selected by the Label Wave method. This measure directly contrasts the performance the Label Wave method with that of the ground-true best model (Global Maximun) across various settings. The Difference is determined by the following formula:
Difference AGMM ALW
Where:
Aga represents the average test accuracy the model which have maximum test accuracy in the training process.
-ALW represents the average test accuracy of the model selected by the Label Wave method.
This formula provides quantitative measure to determine how closely the model selected by the Label Wave method aligns with the optimal model (i.e., the model with Global Maximum test accu- racy) for each setting. A smaller Difference indicates that the performance of the Label Wave method
near optimal, while a larger Difference suggests a divergence from the optimal performance.
Using the Difference metric, objectively evaluate the performance of the Label Wave method across various settings. As shown in Table 1, the consistent and slight disparities between the mod- els chosen by the Label Wave method and the optimal models (Global Maximum) underscore the effectiveness of the Label Wave method across diverse settings. This showcases the Label Wave method's aptitude pinpointing an appropriate early stopping point for model selection, which aligns closely with or even matches the optimal models.
19

---

Published as a conference paper at ICLR 2024
D ENHANCING EXISTING LEARNING WITH NOISY LABEL METHODS
As shown in Tables and 7, we conducted evaluations using ResNet-18 18 on CIFAR-10 and ResNet- 34 on CIFAR-100 (Krizhevsky al., 2009; He et al., 2016), both trained with 40% Instance- Dependent noise (abbreviated as Ins.) (Xia et al., 2020). Further, as shown in Table 8, we added the different between the test accuracy of the model selected using the Label Wave method and the global maximum test accuracy during the training process when using various learning with noisy label methods with 40% Symmetric noise (abbreviated Sym.) (Van Rooyen al., 2015).
Table 6: Test accuracy (meanzistd) of each method on CIFAR-10 10 (with 40% Ins. label noise).
Method
Val. 10%
Val. 20%
Label Wave
Global Maximum
CE (Rubinstein, 1999)
78.04=0.62%
77.35=0.23%
80.49=0.33%
80.86=10.77%
Taylor-CB (Feng ct al., 2020)
81.30=0. 14%
81.09=0.43%
83.24=0.496%
83.63=0660%
ELR (Liu al., 2020)
87.68+0.325%
86.98=0.22%%
89.59+0.166
90.41+0.25%%
CDR (Xia al., 2021)
85.66++0.43%
84.86-0.254%
87.42++0.51
87.63+0..22%
CORES (Cheng ct al., 2021)
79.45=0.38%
78.77=0.35%
81.22=0..449
81.64=00655%
NLS (Wei ct al., 2022a)
80.61=0.80%
80.25=0.25%
82.36=
82.63= 1.24%
SOP (Liu et al., 2022a)
84.08+0.25%
83.51=0.30%
85.57+0.35%
85.92:+0.15%
Table Test accuracy (mean=std) of each method CIFAR- (with 40% Ins. label noise).
Method
Val. 10%
Val. 20%
Label Wave
Global Maximum
CE (Rubinstein, 1999)
42.59--0.21%
41.82-0.64 0.64%
44.86%-0.62%
45.23++0.11%
Taylor-CE (Feng et al., 2020)
52.42.10.59%
52.53 0.56%
55.52.10.54%
55.74.10.3%%
ELR (Liu et al., 2020)
64.9010..3%%
64.61
66.8510.119
67.33.10.57%
CDR (Xia et al., 2021)
59.82--0100 10%
59.21-0.87%
62.64+0.40%%
63.25++0.47%
CORES (Cheng al., 1)21)
43.26++0.71% %
41.96=0.11%
45.38-+0.13%
46.24-10.09%
NLS (Wei et al., 2022a)
52.22-10.88%
53.26=
55.56.10.39%
56.00-10.25%
SOP (Liu et al., 2022a)
60.831
66.33
69.35
70.23.10.53%
Table Test accuracy (mean=std) of each method on CIFAR (with 40% Sym. label noise).
CIFARIO Method
Label Wave
Global Maximum
(Rubinstein, 1999)
81.61=0.44%
81.7610..30%
Taylor-CE (Feng al., 2020)
85.06=0.30%
85.43-10.37%
ELR (Liu et al., 2020)
90.45_10.52%
90.76.10.770
CDR (Xia et al., 2021)
87.6910.109%
87.80.10.2459
CORES (Cheng et al., 2021
87.7420.13%
87.95:10.21%
NLS (Wei al., 2022a)
83.45=0.19
83.62:10.377
SOP (Liu et al., 2022a)
88.42 88.42.0.38%%
88.82.10.469
CIFAR100 Method
Label Wave
Global Maximum
CE (Rubinstein, 1999)
50.96=0.30%
51.05=t0.33%
Taylor-CE (Feng ct al., 2020)
57.64=0.28%
57.99=t0.30%
ELR (Liu et al., 2020)
65.36=0.39%
66.33:+0.93%
CDR (Xia et al., 2021)
63.34-0.15%
63.54++0.28%
CORES (Cheng et al., 2021)
45.03=0.38%
45.75=00.27%
NLS (Wei al., 2022a)
58.05=0.155
58.32:t0.35%
SOP (Liu et al., 2022a)
68.53=0.30%
68.78++0.27%
20

---

Published as a conference paper at ICLR 2024
E SENSITIVITY ANALYSIS OF k VALUE IN MOVING AVERAGES
In this appendix, we present a sensitivity analysis of the k value used in computing moving averages within the framework of the Label Wave method. This method, detailed in the paper, employs moving averages of Prediction Changes (PC) for early stopping in training models with noisy labels.
This analysis varied the k value, which represents the number of epochs over which the moving average is calculated for the PC metric. We computed the Pearson Correlation Coefficient between the moving average of PC values (denoted as PC)) and test accuracy for each & value. The goal was to determine the impact of different & settings on the relationship between the averaged PC values and the test accuracy. The table below summarizes the Pearson Correlation Coefficients for different k values:
I Value
Pearson Correlation Coefficient
-0.8565
2
-0.9435
3
-0,9637
5
-0.9367
10
-0,9406
E.1 OBSERVATIONS
Strong Negative Correlation: strong negative correlation consistently observed across all & values. This signifies a robust inverse relationship between the moving average of PC values and test accuracy.
Variation in Correlation Strength: The correlation's 's strength varied with different k: val- ues. The strongest negative correlation occurred at 3, while k = and = 10 exhibited slight decrease in correlation strength.
E.2 CONCLUSION
The sensitivity analysis reveals that the selection of l value influences the strength of the correlation but does not significantly alter the overall negative relationship between the moving average of PC values and test accuracy.
21


Reviews:
[{"review_id": "QTioTnlsUJ", "reviewer": "Reviewer_8gHt", "summary": "In the face of label noise, this publication presented an early halting technique using the Label Wave approach. Although the finding in this publication is intriguing, it only makes a little contribution to label noise bias.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This publication presented learning perplexing patterns, a transitional stage in learning with noisy labels.", "weaknesses": "The cifar10/100 dataset is utilized in this studies; however, real-world datasets such as webvision and food101 should be employed to confirm the efficacy of the suggested approach. I'm interested in seeing these outcomes.\n\nIn order to determine whether the suggested method chooses the best classifier, I would like to examine the maximum test accuracy during the training phase. \n\nGiven that the focus of this research is label noise, studies should compare the state-of-the-art techniques currently used for label noise learning, like DivideMix[1], ELR[2], AugDesc[3] and so on.\n\n[1] Li, Junnan, Richard Socher, and Steven CH Hoi. \"Dividemix: Learning with noisy labels as semi-supervised learning.\" arXiv\npreprint arXiv:2002.07394 (2020).\n\n[2] Liu, Sheng, et al. \"Early-learning regularization prevents memorization of noisy labels.\" Advances in neural information\nprocessing systems 33 (2020): 20331-20342.\n\n[3] Nishi, Kento, et al. \"Augmentation strategies for learning with noisy labels.\" Proceedings of the IEEE/CVF Conference on\nComputer Vision and Pattern Recognition. 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "daVtRF6F72", "reviewer": "Reviewer_LTbz", "summary": "This paper studies an important topic of learning against noisy labels. Specifically, the authors mainly focus on how to automatical detect the transitioning point for early stopping, i.e., from fitting to clean to fitting to noise. There are two proposed key metrics so-called \"stability\" and \"variability\", and and the method uses \"prediction change\" as the mean of detecting early-stop point. The results show that the method can detect the point accurately by showing the test accuracy difference compared to that obtained from the global maximum point.", "questions": "Please address the two major weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "I felt there are multiple strengths of this work.\n* n-depth analysis: This work provides phase 1 to phase 3 analysis to understand how the DNN learns knowledge from noisy data.\n* Well-defined metric: Several useful metrics are proposed: prediction changes, stability, and variability.\n* The paper is well organized and easy to read.", "weaknesses": "There are several weaknesses in this work, which may be useful to polish the paper.\n* **Missing important reference:** I know references that are highly related to this work but unfortunately not mentioned and compared. These two papers also tackled exactly the same point and mentioned similar intuitions on what the best early stopping point is. These papers are worth mentioning and comparing. \n[1] How does early stopping help generalization against label noise, arXiv 2019\n[2] Robust learning by self-transition for handling noisy labels, KDD 2021\n\n* **Unclear setup for practicality:** Detecting an early stop point is very important in the industry, especially when using strong regularization techniques together. For example, in the computer vision domain, using Mixup (or Cutmix, etc), Batch Norm, Dropout, and other architecture-specific regularization (e.g., stochastic depth for Vision Transformers) is a must-need. These kinds of strong regularization obviously change the learning behavior of DNNs like the training and testing curves. For the complete study toward practical methods, these recipes of training should be considered altogether, or theoretical support is needed.\n\nThese two major issues contribute the most when determining my review score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g5N1fHyRD5", "reviewer": "Reviewer_R2TX", "summary": "The paper introduces an interesting method to perform early stopping in case of label noise without needing a validation set. The writing is good and the method provides some insights. However, there are concerns on the applicability of the method and the experiment set up is insufficient to evaluate the effectiveness of the method.", "questions": "1. Regarding applicability, my intuition is that the method only works well when there is “significant” amount of “random” label noise in the training set. “Random” is because the method relies on that the model fits simple patterns first and then learn random patterns from the noise. What if the label noise also only include simple patterns? E.g. black donkeys are mostly labeled as horses? The method also requires significant amount of label noise so that the model prediction fluctuate to a degree to be detected by the method. This is also reflected by the fact that experiments only considers >20% noise.  Can the authors provide more insights into this through discussion or experiments?\n2. Only datasets with synthetic noise are considered. How does the method work on real datasets with real-world noise?\n3. The baselines seem to use a noisy validation set, and evaluation is done on a clean test set. It makes more sense split the clean test set to create a clean validation set or to create a clean validation set from the training set. This is because we want to ensure the validation set has a same distribution as the test set, i.e. to be clean.\n4. What happens if the amount of label noise is less than 20%? Including label noise from 0-20% can help better understand the method.\n5. In the motivation of not using a validation set is that using a validation set reduces training set size and thus decreases performce, by this motivation, the method is targeting domains with limited amount of training set. How small the dataset size should be when it’s preferred to consider this method? Could you provide some analysis on this?\n6. In order to show the method’s effectiveness, more datasets from diverse domains should be considered.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The method is interesting and has some insights.\n2. The paper is well written, and everything is presented nicely. \n3. Good results are shown on certain noise datasets.", "weaknesses": "1. My biggest concern is the applicability of the method. Currently it’s not clear the method works well on what scenarios. \n2. Experiment evaluation is far from sufficient and the setup can be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bprylPlM8U", "reviewer": "Reviewer_D1rG", "summary": "Early stopping is one of the most prevalent approaches to select model. However, it requires additional validation data. This paper proposes a new method to early stop without using the validation data empirically.", "questions": "- It may not work well for extreme cases, e.g. class imbalanced dataset. Any solution for those settings? If not, some assumptions could be specified for the setting.\n- What is the difference between the previouse studies'[1,2] findings and what authors propose in section 3.2 (fitting\nmislabeled examples impairs the overall model’s fitting performance)?\n- Will PC monotonically decrease before its local minima? In other words, are there no fluctuations? or should we need some threshold?\n- How will this pattern change when utilized with additional regularizations e.g. data augmentation? Will it be consistent or will it flutuate before it goes to local minima?\n- For Table 1 and Table 2, which algorithm is utilized (just Cross Entropy?)? If utilized with several algorithm managing noisy labels, how much different between best and label wave?\n- Want to see result on more noise condition for table 3 and 4.\n- How about on real noise, e.g. Clothing1M?\n- Will this criterion fit to another task, e.g. semantic segmentation?\n\n[1] Wei, J., Liu, H., Liu, T., Niu, G., Sugiyama, M., & Liu, Y. (2022, June). To Smooth or Not? When Label Smoothing Meets Noisy Labels. In International Conference on Machine Learning (pp. 23589-23614). PMLR.\n\n[2] Cheng, H., Zhu, Z., Li, X., Gong, Y., Sun, X., & Liu, Y. (2020, October). Learning with Instance-Dependent Label Noise: A Sample Sieve Approach. In International Conference on Learning Representations.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- It does not require additional validation data for model selection.\n- Experiments on various structures and settings.", "weaknesses": "- What authors proposed is only supported by the empirical result.\n- Following the question 2, more discussions may be needed for the related previous researches.\n- Since the metic is moveing average over k epochs, sensitivity analysis over k is needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)