Adaptive Textual Label Noise Learning based on Pre-trained Models
Authors: Shaohuan Cheng
Wenyu Chen
fu Mingsheng
Xuanting Xie
Hong Qu
Venue: EMNLP 2023 Findings (2023)
Keywords: learning with noisy labels
label noise learning
pre-trained models
text classification
TL;DR: We develop an adaptive textual label noise learning framework based on pre-trained models, which can handle various noise scenarios well, including those with a noise range of up to 40% and mixed noise types.

Abstract: The label noise in real-world scenarios is unpredictable and can even be a mixture of different types of noise. To meet this challenge, we develop an adaptive textual label noise learning framework based on pre-trained models, which consists of an adaptive warm-up stage and a hybrid training stage. Specifically, an early stopping method, relying solely on the training set, is designed to dynamically terminate the warm-up process based on the model's fit level to different noise scenarios. The hybrid training stage incorporates several generalization strategies to gradually correct mislabeled instances, thereby making better use of noisy data. Experiments on multiple datasets demonstrate that our approach performs comparably or even surpasses the state-of-the-art methods in various noise scenarios, including scenarios with the mixture of multiple types of noise.

Adaptive Textual Label Noise Learning based on Pre-trained Models
Shaohuan Cheng, Wenyu Chen, Mingsheng Fu, Xuanting Xie, Hong Qu"
School of Computer Science and Engineering, University of Electronic Science and Technology of China
shaohuanchengêstd uestc. edu. cn, cwy@uestc. edu. cn, fms@uestc.edu.c x624361 380@outlook com, hongquêuestc. www.:uuuuusteeeu.crr....
Abstract
"Game" and "Entertainment" On the other hand, IDN assumes that label noise is dependent on the instance, which simulates the confusion caused by the characteristics of the instance. For example, a piece of news containing the phrase "played on a pitch as slow as a bank queue" may be misclassified as Economic news instead of Sports news due to the specific wording. However, most studies foeus on a particular type of noise. For example, Jin- dal al. (2019) introduces non-linear processing layer to learn the noise transition matrix of CCN, Qiao et al. (2022) designs the class reaassrgularizttion loss according to the characteristic of IDN. How- ever, there is premise for applying these methods, which is that the noise is known and of a single typc.
The label noise in real-world scenarios is unpre- dictable and can even be mixture of different types of noise. To meet this challenge, we de- velop an adaptive textual label noise learning framework based on pre-traie pre-trained models, which consists of an adaptive warm-up stage followed by hybrid training stage. Specifically, an early stopping method, relying solely on the training set, designed to dynamically terminate the warm-up process based the model's fit level to different noise scenarios. The hybrid training stage incorporates several generalization strate- gies to gradually correct mislabeled instances, thereby making better use of noisy data. Exper- iments on multiple datasets demonstrate that our approach performs on-par with or even bet- ter than the state-of sate-of-theearr methods various noise scenarios, including scenarios with the mixture of multiple types of noise.
In the real-world, the noise scenarios are more complex and involve mixture of multiple noises arising from various factors, such as data ambi- guity, collection errors, or annotator inexperience. Methods that specifically target one type of noise are less effective when dealing with other types of noise. This limitation hinders their applicability in real scenarios where the noise is unknown and variable.
1 Introduction
In recent years, deep neural networks (DNNs) have been successfully applied in many fields (Pouyan- far et al., 2018; Alinejad et al., 2021; Liu al., 2023) and the performance largely depends on well-labeled data. However, accessing large-scale datasets with expert annotation in the real world is difficult due to the significant time and labor costs involved. Instead, the noisy data obtained directly from the real world is often utilized in practical scenarios, even though it inevitably contains some incorrect labels. Thus, research on learning with noisy labels has gained attention in various fields such as natural language processing (NLP) (Jindal et al., 2019; Jin et al., 2021; Wu et al., 2022).
To address the challenges posed by real noise scenarios, we develop an adaptive textual label noise learning framework based models. This framework can handle various noise scenarios well, including different types of noise and mixed noise types. Specifically, our approach begins with an adaptive warm-up stage, then di- vides the data into clean and noisy sets by the correctness statistic of samples, and utilizes dif- ferent generalization strategies on them. In partic- ular, there three key designs in our approach. First, the warm-up stage is designed to automat- ically stop early based on the model's fit level the noise scenario, which effectively prevents the model overfitting erroncous labels especially under IDN scenarios or with a high ratio of noise. No-
pre-trained
There are two main types of label noise in NLP: class-conditional noise (CCN) and instance- dependent noise (IDN). CCN assumes that label noise is dependent on the true class, which can simulate the confusion between similar classes like
"Corresponding author

---

tably, the adaptive warm-up method relies solely on the raw training set, rather than clean vali- dation set, making it more suitable for practical scenarios. Second, unlike previous works (Li et al., 2020; Qiao et al., 2022) that fit GMM (Permuter et al., 2006) on the training losses to separate data, the data is separated according to the correctness statistic of each sample. The correctness statistic is accumulated by assessing the consistency between the model's predictions and the given labels during the warm-up stage. This prolonged observation pro- vides more accurate grounds for data partitioning, Third, a linear decay fusion strategy is designed to gradually correct the potential wrong labels to generate more accurate pseudo-labed by adjusting the fusion weights of the original labels and the model outputs.
(Liang et al., 2022). Loss correction tries to reduce the effect of noisy labels during training by adding regularization item in loss, designing robust net- work structure for noisy label and so on. For exam- ple, Wang et al. (2019) adds a reverse cross-entropy term to the traditional loss to reduce the disturbance brought by noise. ELR (Liu et al., 2020) adds a regularization term prevent the model from mem- orizing the noisy labels because it would not be fitted the early training stage. Sample selection divides the data into clean and noisy subsets, and uses different methods for different subsets. For example, Co- Co-Teaching (Han et al., 2018) maintains two networks. During training, the two networks respectively pick out some small-loss samples clean data for each other to learn. DivideMix (Li et al., 2020) uses Gaussian mixture model to sepa- rate clean and noisy samples. The noisy samples are treated unlabeled data, whose pseudo-labels are generated by the model. Finally, Mixmatch (Berthelot et al., 2019) method is adopted for mixed training on clean set and noisy set.
We conduct extensive experiments on four clas- sification datasets, considering different types of noise: class class-coniiiinaal noise, denende noise, and a mixture of multiple noises. To the best of our knowledge, previous methods have not explored such mixed noise. The experimental re- sults demonstrate that our method surpasses isting general methods and approaches the perfor- mance of methods specifically designed for partic- ular noise types in different noise settings. Our contributions can be concluded as follows:
Labels Noise Learning in NLP. The above works mainly focus on vision tasks, the researches on textual scenarios are relatively fewer. Jindal et al. (2019) and Garg et al. (2021) add additional noise modules based on lightweight models such as CNN and L.TTM learn the probability of noise trans- fer. (Liu et al., 2022: Tänzer al., 2021; Zhu et al., 2022: Qiao et al., 2022) conduct research on pre-trained models and find that pre-trained models demonstrate superior performance in noise learning compared to ever, most works focus on one certain type of noise such as CCN. Qiao et al. (2022) studies both CCN and IDN, but still conducts experiments in settings where the type of noise is known and designs spe- cific regularization loss for IDN.
We design an early stopping method for fine- tuning the pre-trained models, which adapts to various noise scenarios and datasets and achieves near the best test accuracy by relying solely on training set.
models. How-
We develop an adaptive noise learning frame- work based on pre-trained models, which can make good use of different types of noisy data while effectively preventing the model from overfitting erroneous labels.
Few works have focused on general methods of label noise in NLP. Zhou and Chen (2021) de- velops a general denoising framework for infor- mation retrieval tasks, which reduces the effect of noise by adding a regularization loss to samples whose model predictions are inconsistent with the given label. Jin et al. (2021) proposes an instance- adaptive training framework to address the problem of dataset-specific parameters and validates its ver- satility across multiple tasks. However, these meth- ods rely on additional components, such as supple- mentary structures or auxiliary dataset, which lim- its their practicality. In contrast, our method relies
Experimental results of various noise settings show that our approach performs comparably or even surpasses the state-of-the-art methods in various noise scenarios, which proves the superiority of our proposed method in practi- cal scenarios.
2 Related work
Universal Label Noise Learning. Label noise learning methods can be divided into two groups: loss correction and sample selection methods

---

Tree
Chusenticorp
3.5
14-
X
Model
0.
Model
Modei
0.4
B BOBBO
18. motty
SEOUGE comecaes Saatanee
Stop
MOTA
15
Model
Epats
Fusthersts to acrage adisnaed lassan teanng
3.5
Ntxcp
6.5
Hybrt training
6.
1.0
tybidd taeitgg
Oflipp Bi D'Doug Dusgp)
Train acc
Test act
MOTA
Figure I: The overall diagram of the proposed method
(a) 40% Asym
(b) 40% IDN
solely on single network and the noisy training data, making it superior in terms practicality.
Figure 2: The memorization observation of different noise cases. Label recall on top figures, train and test accuracy on bottom figures. MOTA is short for the maximum obtainable test accuracy. More cases can be found in Appendix A.
3 Methodology
Problem Definition. Without loss of generaliza- tion, we take text classification as example. Given noisy training dataset Î (x, Y)
an adaptive early stopping condition is involved to terminate the warm-up training before overfitting noisy data. The details of our warm-up stage are given as follows.
{z b the one-hot label ii associated with sample H, is probably wrong. Our goal is learn a classification model p(y|a;; O, from the noisy dataset D. which generalizes well on clean test data. Specifically, the classification model p(y|æ; 0, () consists of pre-trae tre-trained encoder and classifier with parameters 0 and <. respectively.
Training Data. We directly use the raw noisy dataset D warm up the classification model and determine when to stop early.
Learning objective. The standard cross-entropy loss is used to warm up the model:
Overview of the Proposed Method. To address various noise types, we propose general learning method consisting of the warm-up training stage and the hybrid training stage, and the overall di- agram is shown in Figure I. The classification model p(y|a;; A, () is first trained by the raw data in the warm-up stage to form the initial classifica- tion ability. During the warm-up stage, we also maintain correctness statistic to justify whether the model begins to overfit the noisy data. Once there is sign of overfitting to noisy data, we will stop the warm-up, and move on the subsequent hybrid training stage. During the hybrid training stage, the raw data is divided into the clean set and noisy set according to the correctness statistic, and then further train model p(y|æ;; A, () by applying different training strategies to the clean set and the noisy set respectively.
Lwarm Σ (og(p(()) (1)
Overfitting regarding noisy data. Our stopping
condition is based on the following Assumption and Assumption 2.
Assumption 1: As learning progresses, the clean data is fitted faster than the noisy data. We em- pirically demonstrate that the clean samples can be recalled earlier than the noisy samples during learn- ing (see Figure 2), regardless of the noise types. This mainly caused by the memorization effect (Arpit et al., 2017) which means DNNs tend learn simple patterns before fitting noise. As a result, the whole learning process can be roughly divided into Clean Rising (CR) phase (where most clean samples are quickly learned) and Noisy Ris- ing (NR) phase (where the noisy samples are fitted slowly), which are differentiated by backgrounds in Figure 2.
3.1 Adaptive warm-up
The goal of this warm-up stage is to obtain a clas- sification model that fits clean data well but not noisy data. As shown in Figure 2, however, the optimal warm-up time may vary significantly for different noise scenarios. To meet this challenge,
Assumption 2: The prediction regarding noisy data tends to swing between the true label and the

---

Clean Sanple
Naiiy Sumple
should increase fast since the model fits clean sam- ples quickly. In the NR phase, ratio' should stay within a certain range for an extended period be- cause the noisy samples are difficult to fit (Assump- tion 2).
475.67
10h
85% 11
As a consequence, if ratio' stays within a range for times, we can assume that the learning pro- cess has entered the NR phase. To approach the optimal stopping point, we continue to warm up the model until a certain improvement in the training accuracy. The magnitude of improvement is set to be a fraction p1 of the current remaining accuracy. The pseudo-code for adaptive warm-up process is shown in Appendix B. Note that this adaptive warm-up process is suitable for different noise sce- narios due to Assumption I and Assumption 2 can be widely satisfied by different noise types.
Figure 3: Examples of the model output during train- ing under different noise settings. The given label of clean samples is true. More examples can be found in Appendix A.
noisy label. It originates from another memoriza- tion phenomenon regarding noisy samples(Chen et al., 2021). The results in Figure 3 also demon- strate that the prediction for the noisy samples ex- hibits higher level of inconsistency with the given label due the activation of the true label.
Correctness statistic. According to these two assumptions, we update the correctness statistic of training set at intervals to judge whether the model begins to overfit the noisy data. Specifically, for given sample T; with label It. its correctness coefficient m is:
3.2 Hybrid training
To further leverage the underlying values of noisy samples, we propose hybrid training method ap- plying different training strategies to clean samples and noisy samples respectively.
Data. Based on the correctness statistic M {mā] (mf) J (assuming l' is the stopping time of the warm-up stage), the whole training set {( di) can be divided into the "clean" set De and "noisy" set D, as follows.
mf = =Err
(2)
obtained by,
1, if arg max maxpp (z,30, () arg max of REK
D. - DX=(X,, Yc) = {1x, Di m 2 4. Da = (Xn: I,) = {( if < l},
r/
KE K
(5)
11 else.
(3) where the correctness r indicates whether the pre- diction of sample x; is consistent with the given label at moment t, and mf is the statistic of correct- ness in given time range.
where is the Np2-th largest correctness statis- tic value of M because larger number indicates that the sample is fitted earlier and has higher probability of being a clean sample. P2 is a given percentage, which set to 20%.
The higher the number of correct predictions compared to incorrect predictions for sample, the higher the degree to which the model fits that sample. Intuitively, positive m indicates x; has been fitted by the model at moment t. Furthermore, we have a proportion ratio'.
Pseudo-labeling. To minimize the side-effects of the noisy labels, we regenerate labels for both the clean set D. and noisy set Dn through the pseudo- labeling method combining the original labels and the prediction of the classification model. Note that there may inevitably be some noisy samples in set D. albeit fewer than in D,,.
1 ratio = N Σ T(ml > 0),
(4)
For each sample (2r; Øi) in clean set De. the corresponding pseudo- &J is obtained by,
which indicates the sample fitting level for the whole dataset with N samples.
01 -:w 04 (1 w/) Mla;0,9) (), where the weight wa decays linearly with the train- ing step t, which is calculated by,
(6)
Early stopping condition. Through observing the change of ratio', we can determine whether the learning process has entered the NR phase. Ac- cording to Assumption 1, in the CR phase, ratio
t
W = 1 (1 61) T
(7)

---

Datasets Experiments are conducted on four text classification datasets, including Tree (Voorhees et al., 1999), Agnews (Zhang et al., 2015), IMDB (9) (Maas et al., 2011) and Chnsenticorp (Tan and Zhang, 2008), where Chnsenticorp is Chinese (11) dataset and the rests English datasets. The statistics of datasets are presented in Table 1, where where T indicates the all training steps of this stage, wh decays from to d1 due relatively reliable labels in the clean set De. Likewise, for each sample in noisy set D,, its pseudo-labels ÿt are obtained by,
Datasets
Clusses
Traning
Testing
Type
Tree
6
5,452
500
Question
Agnews
120,000
7,600
News Topic
IMDB
25,000
25,000
Movic Review
Chnsenticorp
10,430
1,200
Hotel Review
Table 1: The statistics of datasets
" = 82 (1 7)), = wh: Vs (1 w 0(a;;,,))
(8)
R-Drop. Since the prediction the model is fused in pseudo-labeling, it needs to be as accurate as possible. Therefore, the R-Drop strategy (liang et al., 2021) is applied on noisy samples Xn to promote the model to have consistent output dis- tribution for the same input. R-Drop minimizes the divergence between two distribu- tions predicted by the model with dropout mecha- nism for the same sample:
where wn decays from 82 to 0 due to most labels in Dn are incorrect.
By linearly decaying the weight of original la- bels, a good balance is achieved between leverag- ing the untapped label information and discarding noise. As the accuracy the classification model's predictions continues to improve, the generated pseudo-labels will be closer to the true labels.
Kullback-Leiin
M
Furthermore, for each pseudo-label i we adopt the sharpen function to encourage the model generate low entropy predictions, i.e., Út I|. where 7 is the temperature parame-
Ldrop
DxL (x),))())
2
1=1
+DKL 0
M 1/r 1/rl
(14)
ter and ||-11, is nnnom. Finally, the new whole set D = {(x,, 9 reformed by clean and noisy
where M is the number of noisy samples X Pi (2g; 0) and (2 0) are two predictions of z4.
sets.
Learning objective. The total loss is:
Mixup. To enhance the generalization of the model, mixup technique (Zhang et al., 2017: Berth- elot al., 2019) is adopted introduce diverse and novel examples during training. Different from mixing pictures directly in vision tasks, the text input cannot be directly mixed due to the discrete- ness of words. Like previous work (Berthelot et al., 2019; Qiao et al., 2022) in NLP, the sentence pre- sentations encoded by pre- preitrained encoder are used to perform mixup:
C Emix + BL
(15)
where is the hhper--yrammnn parameter KL loss, which is set to 0.3 in experiments.
Experiments
4.1 Experimental settings
= Beta(a, a))
X' = max(A, - A)
(10)
h' = Xp(en; + (1- {))(z;;00
y' Xoo + (1- X²}
(12) the training set of Chnsenticorp merger of the original training and the validation set.
where a is the parameter of Beta distribution, p(z; 8) is the sentence embedding which corre- sponds to "[CLS]" token. (x; Oi) and (æy, 05) are randomly sampled in the corrected set D.
Noise types The following types of noise are in- jected to standard datasets:
Class-condittonal noise: We choose typical symmetric (Sym) and asymmetric (Asym) noises in various class-conditional noise to conduct exper- iments. Symmetric noise flips certain percentage of labels in a given category into other categories uniformly. Asymmetric noise flips labels between given similar class pairs. IMDB and Chnsenticorp
The mixed hidden states { and targets ( used train the classifier by apply- ing entropy loss:
N Lmic Σ
Eolog(p(H(:C))
(13)

---

Noise Type
Sym
Asym
IDN
(Sym & Asym)
even Asym & IDN
(all thros)
Datasct
Agnooss
Agnows
IMDB
Chn
Agnows
Thoe
IMDB
Apricws
Noise Ratio
40
-00
40
20
40
20
40
20
40
20
40
92.68
92.80
84.85
95.92
93.28
61.28
83.06
92,09
90.87
BERT-FT
Last
87.40
80.92
76.60
71.30
64.07
66.82
69.26
68.67
67.03
95.16
85.64
85.79
72.46
85.63
60.68
90.78
65.44
CT
Best
94,16
92.98
90.20
02.61
81.35
91.31
72.72
96.16
94.30
80.87
85.01
92.37
91.23
Last
88.28
85.75
77.04
91.17
61.85
06.58
70.01
70.09
09.06
94.88
58.08
86.22
72.68
87.53
08.95
91.21
88.21
Best
94.68
95.06
92.64
92.88
85.70
X3
75.65
72:08
96.24
94.64
80.58
92.65
83.63
92.30
90.94
Last
93,48
91.68
85.24
90.73
78.28
81.91
70.18
68.77
09.01
96.16
93.52
87.15
76.10
90.28
79.58
91,74
90.11
Best
51.04
92.91
95.32
93.22
67.41
59.02
84,09
80.74
34.02
95.64
94.16
59.38
84.34
91:11
87.70
92.11
91.13
SeltMix
Last
93.56
92.71
94.96
92.95
86.19
83.45
83.15
79.99
83.52
94.28
93.60
87.88
82.67
92.03
56.66
86.55
90.24
Best
94.72
93.27
96.32
93,56
89.13
91.65
85.40
80.07
84.25
96.48
94.3 46
89.05
K3.47
93.23
88.47
92.65
91.33
Last
94.20
92.33
96.00
91.06
88.21
80.00
09
84,76
75.42
84.03
96.16
93.68
57.87
81.58
88.36
83.42
91.74
90.25
Table 2: Results (%) of five runs on Trec, Agnews, IMDB and Chasenticorp under different noise settings. The bolded results means the highest among all best scores and the underlined results means the highest among all last scores. Chn is short for Chnsenticorp. means some nyper-parameters are adjusted according to the type noise and the composition ratio of mixture noise.
are binary classification datasets, so their symmet- ric and asymmetric noises are the same.
of wrong samples. SelaMix(Qiao et al., 2022): SelfMix first warms up the model and then uses GMM to separate data to perform semi-supervised self-training, and de- signs the normalization method according to the characteristics of IDN, resulting in significant per- formance improvement compared other methods lacking specific designs.
mnstance-dependens noise: Follow (Qiao et al., 2022). LSTM classifier is trained to determine which sample features are likely to be confused. The labels of the samples closest the decision boundary are flipped to their opposite category based on the classifier prediction probability. All datasets except Tree are used because of its small sample size and uneven categories. Main experi- ments only show results for single type of noise with ratio of 40%, results for other ratios can be found in the Appendix C.
For fair comparison, the backbone model of each method is the same, including pre- pre-trained encoder BERT' and two-layer MLP. The training time of all methods except SelfMix is set to 6 epochs and their hyper-parameters are the same under different noise settings. For SelfMix, we follow the instruc- tions of its paper to set different hyper-parametee for different datasets and noise types. Specially, the warm time for symmetric and asymmetric noise is 2 epochs, the warm time for instance-dependen noise is 1 epoch, and the semi-supervised self- training time is 4 epochs. The average results are run five times the code base provided by Qiao et al. (2022).
Mixture of multiple noises: We mix different types of noise to verify the ability of the algorithm to deal with complex and unknown scenarios. For datasets with only two types of noise, including Trec, IMDB and Chnsenticorp, we mix the two noises evenly. Specifically, Sym and Asym are evenly mixed on Trec, Asym and IDN are evenly mixed on IMDB and Chnsenticorp. For Agnews, we mix three noises unevenly for a larger challenge. The result of evenly mixing of two types of noise on Agnews is in Appendix C.
4.3 Parameter settings
The same hyper-parameters baselines remain un- changed, where the maximum sentence length is 256, the learning rate is 1e-5, the drop rate is 0.1, the size of middle layer of MLP is 768 and opti- mizer is set to Adam. Additionally, the temperature T is 0.5, a of Beta distribution is 0.75.
4.2 Baselines
BERT-FT (Devlin et al., 2018): the benchmark classification model without special methods, di- rectly trained on noisy data.
Co-Teaching (CT for short) (Han et al., 2018): a well-known noise learning method, which main- tains two models simultaneously and lets each model select clean samples for the other to train. ELR (Liu al., 2020): ELR designs a regulariza- tion term to prevent the model from memorizing noisy labels by increasing the magnitudes of the co- efficients on clean samples to counteract the effect
There are some specific parar-parrmeters in our method. Specifically, in warm up stage, the inter- val of monitoring training set is 1/10 epoch, the converge range e is 0.01, the times 1) is 3 and the
'the weight of BERT for English task is initialized with hert-hasc-uncased, and the wight for Chinese task is initialized with wwweese-hhrt-www--e-. of HIT.

---

fraction pt is 0.1, the batch size is 12. In hybrid training stage, the fraction p2 to divide data is 0.2, the threshold 01 is 0.9 and 82 is 0.4, and the loss weight / is 0.3. The batch size of clean set in hy- brid training stage is 3 and the batch size of noisy set is 12 to match the ration P2 of their numbers.
Dataset
Tree
Chaseuticorp
Noise Type Method/Raati
Asym
IDN
10
40
Clean
99.
94.89
95.71
75.57
Ours
Best
96.68
96.32
84.25
Last
96.36
96.00
93.24
84.03
Best
90.08
93.53
83,97
w/o linear decay fusion
Last
95.48
89.80
92.34
81
Best
96.32
95.16
93.78
74.89
The duration of the warm up stage varies under different noise settings. The hybrid training stage lasts for 4 epochs to compare with SelfMix. For generality, the parameters used in all cases are same as above.
w/o correctneas statistic
Last
96.24
94.76
92.33
68.00 68
Best
96.32
96.20
94.17
79.67
w/o mixup
Last
95.72
95.96
90.22
74.22
Best
96.04
95.72
93.30
76.58
r-drop
Last
95.40
95.20
75.08
64.53
Best 94.20
93.64
94.33
83,60
I epoch warm-up time
Last
93.44
93.24
90.97
83,60
Best
96.36
95.36
93.80
80.65
4.4 Main Results
2 epoch warm-up up time
Last
96.16
95.12
92.55
78.20
Best
96.60
96.80
93.30
77.15
Table 2 demonstrates the main results of our method and baselines.
3 epoch warm-up time
Last
96.36
96.52
90.33
75.87
Best
96.52
96.44
93,47
70.35
4 epoch warm-up
Last
96.24
96.20
88.98
69.32
Clean
98.94
92.40
94.76
Baselines BERT-FT is highly affected by noise, with performance and stability decreasing sig- nificantly as the complexity noise increases. CT and ELR demonstrate strengths in handling simple noise scenarios, such as Trec with 40% symmetric noise and a mixture of symmetric and asymmet- ric noise. However, they perform poorly under complex noise settings, such IDN noise. In contrast, SelfMix excels in cases involving IDN due to its design of class-rogularizztion loss and re- duced warm-up time. Among all the methods, only SelfMix needs to adjust specific paper--prameterss based on the dataset and noise type.
separate with GMM
Best
96.16
93.28
92.82
05
Last
95.68
93.16
91.58
79.50
Ablation study results in terms of test accuracy (%) on Trec and Chnsenticorp under different noise. The highlighted rows denotes the ratio of correct labels in the separated clean set.
Table 3:
5.1 Ablation experiment
To verify that each component contributes the overall approach, we remove each component sep- arately and check the test accuracy. To remove linear decay fusion, the original labels of clean set are kept and the pseudo-labels of noisy set are gen- crated by the model. For the design of correctness statistic, the relevant parts including early stopping and linear decay fusion are removed and replaced by normal training. The standard cross entropy loss are directly applied all samples and the corre- sponding pseudo labels to remove mixup operation. For R-Drop, the KL-divergence term is climinated. The first part of Table 3 shows each component is helpful to the final performance. Correctness statis- tic and R-Drop are more critical to complex setting (i.e. 40% IDN), because the model is greatly af- fected by this noise. And due to the introduction of linear decay fusion, R-Drop becomes more impor- tant in the aspect of keeping the model stable.
Our method In contrast to the baselines, our method consistently performs well across all noise scenarios. With simple CCN setting, our method exhibits further improvements in performance com- pared to well-performing baselines like ELR. With IDN setting, our method outperforms general base- lines and even surpasses SelIMix in certain cases, such Agnews and Chnsenticorp. Moreover, when confronted with mixed noise, our method consistently achieves top near-top results while maintaining stability. conclusion, our approuch demonstrates superior performance and adaptabil- ity various noise scenarios, making it more prac- tical for real-world scenarios.
5 Analysis
5.2 Analysis of early stopping
In this section, some experiments are designed to make a more comprehensive analysis of our pro- posed method. The main results are shown in Table 3 and the correspond analyses are as follows. More analyses can be found in Appendix D.
Stopping warm up properly is important for adapt- ing various noise cases and datasets. To ver- ify this view, experiments with different warm-up times are performed. The numbers of war-up epoch are set to (1,2,, 4) and the results are shown in the

---

Trec
Chnsenticorp
MethodssDatasees
Trec
Chnsenticorp
Agnews
BERT-FT
3min
22min
2h 17min
0.
SelfMix
Smin
42min
4h 40min
0.0
Ours
5min
43min
4h 48min
0.1
0.-
MOTA
Tee set
Table 4: The time cost example of different methods under the same noise setting.
(a) 40% Sym
(b) 10% IDN
0.8
value in the correctness statistic the clean set. different from previous noise-learning works (Li al., 2020; Qiao al., 2022; Garg et al., 2021) where the sets are separated by GMM or BMM. To make comparison, GMM is used in hybrid training stage and divide data at each epoch. In addition to the results, the ratios of correctly separation (i.e. how many samples in clean sets are truly clean) are also listed in Table 3, and highlighted with the gray background. As shown, the right ratio of our method is slightly higher than that of GMM, and the performance is better especially under the higher noise ratio settings.
0.1
0.2
(c) 40% Asym
(d) 40% IDN
Figure 4: The instances of early stopping under different noise scenarios.
second part of Table 3. We can observe that no one fixed warm-up time is suitable for all situations. For simple cases such asymmetric noise, warm up for epochs is beneficial. For complex cases such as instance-dependent noise, warm up for I epoch is enough. But adaptive warm-up works better than fixed times on processing all cases. Therefore, find- ing appropriate stopping point according to differ- ent noisy training datasets may be a good strategy of noise learning based pre-trained models.
5.4 Computational cost
Compared to previous noise learning approaches that include normal warm-up phase (Han et al., 2018; Qiao et al., 2022), our method involves additional cost incurred by inferring the entire train- ing set at intervals during the adaptive warm-up phase. This cost is directly proportional to the num- ber of training set samples and the frequency of inference. To strike a balance between monitor- ing and cost, we set the interval to 1/10 epoch in our experiments. Additionally, during the hybrid training stage, reduce the cost of inferring the training set and fitting the GMM compared to Self- Mix. Table provides an example of the time costs of BERT-FT, SelfMix and our method under the same settings, with all methods trained on single GeForce RTX 2080Ti.
To check whether our early stopping method finds the stopping point close to the maximum ob- tainable test accuracy (MOTA), we draw some find- ing instances compared with other two heuristic approaches. including stopping through noise ratio (Arpit et al., 2017) and clean validation set (we use test set for validation here). As shown in Figure 4, the estimated start and end points of early stop- ping (ES) are close the MOTA under different noise settings. And in some cases ES is closer to the MOTA than stopping through noise ratio (Ra- tio) especially under IDN, Because IDN easier fitted by deep models, which resulting in high train- ing accuracy at early stage. Compared with the two methods relying on additional conditions, our method only relies on the original noisy training set, which is more adaptable to the real world.
Conclusion
Based on the unknown and complex nature of noise, we propose a noise learning framework based on pre-trained models that can adapt to various tex- tual noise scenarios. This framework automatically stops the warm-up process based on the magnitude and complexity of the noise prevent the model from overfitting noisy labels. To further leverage mislabeled data, the linear decay fusion strategy is combined with mixup and R-Drop to improve per- formance while maintaining stability. Experimen-
5.3 The design of correctness statistic
During warm up stage, correctness statistic is maintained to judge the time of early stopping. In addition, is used to separate the training data into clean set and noisy set for training in hybrid stage. We directly select the top 20% samples of the high-

---

tal results demonstrate that our method achieves performance comparable to state-of-the-ar in all settings within common noise range.
bat instance-dependent label noise. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 35, pages 1144--11450.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and
Limitations
Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing, arXiv preprint
We would like to introduce a versatile noise frame- work that adapt to various noise scenarios and have conducted extensive experiments across dif- ferent simulated scenarios to evaluate the perfor- mance. However, it is crucial to acknowledge that we didn't tt experiment on real textual noise scenar- ios. If the noise learning method can be verified in real industrial datasets, it will be more convincing, Furthermore, due to the necessity of monitoring the training set during the warm-up stage, the overall training time of our method tends to be longer com- pared other approaches, especially when dealing with large datasets like Agnews. Resolving this issue or exploring alternative approaches to reduce training time is a direction that requires further investigation, which we leave to future work.
Siddhant Garg, Goutham Ramakrishnan, and Varun Thumbe. 2021. Towards robustness to label noise in text classification via noise modeling. In Proceed- ings of the 30th ACM International Conference on Information & Knowledge Management, pages 3024- 3028.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu. Ivor Tsang, and Masashi Sugiyama. 2018. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
Lifeng Jin, Linfeng Song, Kun Xu, and Dong Yu. 2021. Instance-adaptive training with noise-robust losses against noisy labels. In Proceedings of the 2021 Conference Empirical Methods in Natural Lan- guage Processing, pages 5647-5663, Online and Punta Cana. Dominican Republic. Association for Computational Linguistics.
Acknowledgements
This work was supported by the Science and tech- nology support program of Sichuan Province under Grant 2022Y1
Ishan Jindal, Daniel Pressel, Brian Lester, and Matthew Nokleby. 2019. An effective label noise model for DNN text classification. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 3246-3256. Minneapolis, Minnesota. Association for Computational Linguistics.
References
Ashkan Alinejad, Hassan S. Shavarani, and Anoop Sarkar. 2021. Translation-basedd supervision for pol- icy generation in simultancous neural machine trams- lation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1734-1744, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Junnan Li, Richard Socher, and Steven CH Hoi. 2020. Dividemix: Learning with noisy la- bels semi-supervised learning. arXiv preprint arXiv:2002.07394
Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kan- wal. Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. 2017. A closer look at memorization in deep networks. In Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, pages 233-242. PMLR.
Xiaobo liang, Lijun Wa, Juntao L.i, Yue Wang. Qi Meng, Tao Qin, Wei Chen, Min Zhang, and Tie-Yan Liu. 2021. R-drop: Regularized dropout for neural net- works. Advances in Neural Information Process- ing Systems, volume 34, pages 10890-10905. Curran Associates, Inc.
Xuefeng Liang, Xingyu Liu, and Longshan Yao. 2022. Review-a a survey learning from noisy labels. ECS Sensors Plus. 1(2):221401.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raf- fel. 2019, Mixmatch: A holistic approach to semi- supervised learning. In Advances in Neural Informa- tion Processing Systems, volume 32, Curran Asso- ciates, Inc.
Bo Liu, Wandi Xu, Yuejia Xiang, Xiaojun Wu, Lejian He. Bowen Zhang, and Li Zhu. 2022. Noise learning for text classification: A benchmark. In Proceed- ings of the 29th International Conference on Com- putational Linguisties, pages 4557-4567, Gyeongju, Republic of Korea. International Committee on Com- putational Linguistics.
Pengfei Chen, Junjie Ye, Guangyong Chen, Jingwei Zhao, and Pheng-Ann Heng. 2021. Beyond class- conditional assumption: A primary attempt to com-

---

Sheng Liu, Jonathan Niles-Weee Narges Razavian, and Carlos Fernandez Sanaandez-Grand. 2020. parly-learning reg- ularization prevents memorization of noisy labels. In Advances in Neural Information Processing Systems, Curran Associates,
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. 2017. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:17100 19.09412.
volume 33, pages Inc.
Xiang Zhang. Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text clas- sification. In Advances in Neural Information Pro- cessing Systems, volume 28. Curran Associates, Inc.
Wanlong Liu, Shaohuan Cheng, Dingyi Zeng, and
Qu Hong. 2023. Enhancing document-level event argument extraction with contextual clues and role relevance. In Findings of the Association for Compu- lational Linguistics: ACL 2023, pages
Wenxuan Zhou and Muhao Chen. 2021. Learning from noisy labels for entity-centric information extraction. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 5381-5392, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Andrew Maas, Raymond E Daly, Peter Pham, Dan Huang, Andrew Y Ng. and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the associ- ation for computational linguisties: Human language lechnologies, pages 142-150.
Dawei Zhu, Michael A Hedderich, Fangzhou Zhai, David Ifeoluwa Adelani, and Dietrich Klakow. 2022. Is bert robust to label noise? a study on learning with noisy labels in text classification. arXiv preprint 290.0720...9371...
Haim Permuter, Joseph Francos, and Ian Jermyn. 2006. A study of gaussian mixture models of color and texture features image classification and segmen- tation. Pattern recognition, 39(4):695-706.
Samira Pouyanfar, Saad Sadiq, Yilin Yan. Haiman Tian, Yudong Tao, Maria Presa Reyes, Mei-Ling Shyu, Shu-Ching Chen, and Sundaraja S Iyengar. 2018. A survey on deep learning: Algorithms, techniques. and applications. ACM Computing Surveys (CSUR), 51(5):1-36.
Dan Qiao, Chenchen Dai, Yuyang Ding, Juntao Li, Qiang Chen, Wenliang Chen, and Min Zhang. 2022. SelfMix: Robust learning against textual label noise with self-mixup training. In Proceedings of the 29th International Conference on Computational Linguis- tics, pages 960-970, Gyeongju, Republic of Korea. International Committee on Computational Linguis- tics.
Songbo Tan and Jin Zhang. 2008. An empirical study of sentiment analysis for chinese documents. Expert Systems with applications, 2424)2262.
Michael Tänzer, Sebastian Ruder, and Marek Rei. 2021. Memorisation versus generalisation in pre-trained language models. arXiv preprint arXiv:2105.00 008888
Ellen M Voorhees, Dawn M Tice, al. 1999. The trec-8 question answering track evaluation. In TREC, volume 1999, page 82.
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jin- feng Yi, and James Bailey. 2019. Symmetric cross entropy for robust learning with noisy labels. In Pro- ceedings of the JEEE/CVF International Conference on Computer Vision, pages 322-330.
Tingting Wu, Xiao Ding, Minji Tang, Hao Zhang, Bing Qin, and Ting Liu. 2022. STGN; an implicit reg- ularization method for learning with noisy labels natural language processing.
Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 7587-7598, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.

---

A Examples of Assumption
Mixture of multiple noises: As shown in Table 7, mixed noise pose challenges, and various base- lines exhibit distinct characteristics. BERT-FT is highly impacted by noise, particularly with a signif- icant drop in the Last score. CT and ELR demon- strate their strengths in handling simple noise sce- narios, such as a mixture of symmetric and asym- metric noise. On the other hand, SelfMix excels in cases involving IDN. In contrast, our method consistently achieves top or near-top performance while maintaining stability across all scenarios. It offers a more flexible and adaptable solution for unknown noise cases.
The more examples of two assumptions are shown in Figure 5 and 6.
B Pseudo-code of Adaptive Warm-up
We summarize the pseudo-code of adaptive warm- up in Algorithm 1.
Experiments
C.1 Detailed Results
The detailed results are given in this section. Table 5 shows the results of different ratios of CCN, Table 6 shows the results of different ratios of IDN, and the results of the mixture of two types of noise on Agnews supplemented in Table 7.
In conclusion, our method gets good enough scores under all noise conditions without knowing the noise type, noise ratio the characteristics of dataset. Besides, the same hyper-parameters are used for all datasets and noise settings. These factors make our approach more practical real scenarios.
Class-conditional noise: With this simple as- sumption, our proposed method basically achieves the best scores across all noise cases in Table 5. All methods can obtain very good performance under a low noise ratio due to the deep model is robust to simple noise. However, there are still some gains be made with our approach in some cases such as the results on Tree and Chnsenticorp with 20% asymmetric label noise. Under high noise ration i.c. 40%, the baselines are affected to a greater or lesser extent, but our method performs well on different datasets and outperforms all methods. Ad- ditionally, we list the results on clean datasets with- out noise. The performance gap between clean data and noisy data are getting smaller by applying our method.
D More Analysis
D.1 Analysis of linear decay fusion
In linear decay fusion, set 61 0.1 to let the weight of label of cleaner data decay from to 0.1, and set 82 = 0.6 to let that of noisier data from 0.6 to O. To understand the role of two thresholds, Table 8 lists some results of fixing one of them and changing the other. The first three rows shows that 8, has little effect on asymmetric noise but limits the performance of IDN. Because IDN is easily fitted by the model, some noisy samples are inevitably assigned cleaner set. It is more appropriate to reduce the weight to small value. As can be observed from the last two lines, small value of & is also detrimental to IDN, because it introduces much noise from noisier set. But large value lets some valuable information to be lost, the trade-off value 0.6 is a good choice. The setting of the two thresholds allows the proposed method to handle complex noise cases and cover possible noise situations, thus making well applicable other datasets or the real world.
Instance-dependent noise: Table 6 shows that the complexity of IDN results in substantial de- crease in performance for general methods CT and ELR, indicating that they cannot cope well with this noise setting. In these experiments, SelfMix gets more best scores, which is related its special designs based on noise type such as reducing the warm-up time, designing rlaualeggulaiizzatio loss and etc. In contrast, our method exhibits strong per- formance in all cases and achieves scores close to or even better than SelfMix without making assump- tions about the type of noise. Particularly notewor- thy are the cases where SelfMix underperforms, such as Trec and Chnsenticorp with 10% noise, whereas our method surpasses even the strongest baseline. This further highlights the versatility and effectiveness of our method across different scenar- ios.
D.2 Stability Analysis
In main experiments, the time of hybrid training is set 4 epochs to fairly compare with SelfMix. The stability of noise learning methods is also an important aspect, and the training time is extended to verify the stability. Since the warm up time is adaptive, we set different hybrid training times.

---

Dataset
Tree
Agnews
IMDB
Chnsenticorp
Noise Type
Sym
Asym
Sym
Asym
Sym/Asym
Sym/Asym 20
Method/Raaio
20
40
20
40
20
40
20
40
20
40
40
No Noise
97.04
94.53
92.39
96.50
Best
96.36
94.12
95.88
90.96
93.85
92.68
94.06
92.80
90.46
84,85
94.08
81.88
BERT-FT
Last
94.92
87.40
92.36
76.60
90.01
80.92
90.78
71.30
81.57
64.07
84.58
66.82
Best
96.60
94.16
96.04
90.20
94.02
92.98
94.19
92.61
90.65
81.35
94.53
91.21
CT
Last
95.32
88.28
94.92
77.04
91.27
85.75
93.58
91.17
84.73
61.85
87.61
66.58
Best
96.60
94.68
96.16
92.64
93.98
93.06
94.30
92.88
90.65
85.70
94.03
91.83
ELR
Last
96.24
93.48
95.24
85.24
93.69
91.68
93.71
90.73
89.52
78.28
92.75
84.91
Best
96.08
94.04
95.76
95.32
93.95
92.91
94.08
93.22
91.35
87.41
94.08
89.02
SelIMix
Last
94.96
93.56
94.64
94.96
90.08
92.71
90.50
92.95
90,43
86.19
89.02
83.45
Best
96.55
94.72
96.68
96.32
94.33
93.27
94.48
93.56
91.55
89.13
94.81
92.65
Ours
Last
96.30
94.20
96.36
96.00
92.80
92.33
92.52
91.06
90.78
88.21
93.85
89.99
Table 5: Results (%) of five runs on Trec, Agnews, IMDB and Chnsenticorp under different ratios of class- conditional noise (CCN, inculding symmetric (Sym) and asymmetric (Asym)). The bolded results means the highest among all best scores and the underlined results means the highest among all last scores. The "No Noise" row shows the results of directly fine-tuning in original clean datasets.
Dataset
Agnews
IMDB
Chasenticorp
Method/Ratio
10
20
30
40
20
40
10
20
30
40
Best
91.90
88.72
84,90
75.84
89.81
84.56
78.48
72.16
93.26
87.76
80.33
73.66
BERT-FT
Last
91.21
86.10
78.67
69.26
88.57
82.21
75.88
68.67
90.66
83.30
74.16
67.03
Best
92.02
89.08
84.52
77.34
89.82
85,86
80.38
72.72
93.90
88.63
82.03
74.06
CT
Last
91.33
85.88
78.26
70.01
88.84
81.85
76.13
70.09
91.56
82.63
77.15
69.06
Best
92.05
88.92
85.04
75.65
90.24
83.70
78.14
72.08
93.75
88.55
80.71
73.57
ELR
Last
91.21
87.10
79.85
70.18
88.32
82.08
73.51
68.77
91.15
82.76
77.90
69.01
Best 91.86
90.17
88.92
84.09
90.50
87.76
83.12
80.74
93.38
91.64
88.58
84,02
SeltMix
Last
89.71
88.63
87,94
83.15
88.97
85.09
80.84
79.99
91.45
88.62
85.95
83.52
Best 92.40
89.93
87.72
85.40
90.43
86.56
83.42
80.07
94.75
90.23
87.63
84.25
Ours
Last 90.77
88.01
83,40
84.76
90.17
84.70
78.07
75.42
93.24
88.63
87.12
84.03
Table 6: Results (%) of five runs on Agnews, IMDB and Chnsenticorp under different ratios instance-denpendent noise (IDN). The bolded results means the highest among all best scores and the underlined results means the highest among all last scores.
Uneven mixture of three noises
Mixed Type
50%
5ym+50% Asym
50%
Asym +50% IDN
Dataset
Trec
Agnews
IMDB
Chnsenticorp
Agnews
Agnews
Method/Ratio
20
40
20
40
20
40
20
40
20
40
20
40
Best
95.92
93.28
93.96
92.83
88.87
81.28
91.86
83.06
91.72
88. 18.12
92.09
90.87
BERT-FT
Last
95.16
85.64
90.20
79.81
85.79
72.46
85.63
69.68
90,26
81.88
90.78
85.44
Best
96.16
94.20
94.17
92.89
88.30
80.87
92.91
85.01
91.91
88.43
92.37
91.23
CT
Last
94.88
88.68
90.76
82.94
86.22
72.68
87.53
68.95
90.58
82.66
91.21
88.21
Best 96.24
94.64
94.08
93.13
88.92
80.58
92.65
83.63
91.73
88.34
92.30
90.94
ELR
Last
96.16
93.52
93.88
91.58
87.15
76.10
90.28
79.58
90.76
86.03
91.74
90.11
Best 95.64
94.16
93,94
93,08
89.88
84.34
92,91
87.70
91.80
88.93
92.11
91.15
*SelfMix
Last
94.28
93.60
90.70
92.98
87.88
82.67
92.03
86.66
85.89
88.70
86.55
90.24
Best 96.48
94.36
94.30
93.39
89 89.05
83.47
93.23
88.47
92.13
89.18
92.65
91.33
Ours
Last
96.16
93.68
93.65
91.39
87.87
81.58
88.36
85.42
90.88
85.65
91.74
90.25
Table 7: Results (%) of five runs on Trec. Agnews, IMDB and Chnsenticorp under different ratios of mixed noise. The bolded results means the highest among all best scores and the underlined results means the highest among all last scores. * denotes the best results chosen from different parameter combinations.

---

Tree
Tred
Chasenticorp
Chnsenticorp
Chnsenticorp
M0TA
CP-
Tia sc:
4006
(a) 40% Sym
(h) 40% Asym
(c) 40% Asym
(d) 10% IDN
(c) 40% IDN
Figure 5: The memorization observation of different noise cases. Label recall top figures, train and test accuracy on bottom figures.
Dataset
Trec
Chnsenticorp
Noise Type
Asym
IDN
Cleas Sample
Noty Sample
Epoch/Ratio
20
40
10
40
Best
96.68
96.32
94.75
84.25
4
Last
96.36
96.00
93.24
84.03
48% A77o
Best
96.80
96.64
94.40
81.98
6
Last 96.36
96.04
89.72
80.68
4m IBN
Best
97.04
96.52
94.22
83,78
Agree Agm
8
Last
96.20
91.84
89.63
81.93
Agres
Best 96.84
96.44
94.75
85.37
4+2
Apera
Last 96.48
96.12
87.43
83.15
Best 96.88
96.52
94.75
85.37
19:10K
4+4
Last
95.84
94.80
90.18
82.93
Figure 6: Examples of model output under different noise settings.
Table 9: Results in terms of test accuracy (%) with different hybrid training times on Tree and Chnsenticorp under different noise.
There are two modes to extend because of the ex- istence of linear decay fusion. The first is that the decay time is extended as same as the training time, which is expressed as uniform number in the ta- ble. The second is that the decay time is fixed
Dataset
Trec
Chusenticorp IDN
Noise Type
Asym
Weight Rhrtshold/Ratio
20
40
40
Best
96.68
96.32
94.75
84.25
(0.1,
Last
96.36
96.00
93,24
84.03
4 epochs and the total training time is extended.
(0.8, 0.6)
Best
96.64
96.52
93.97
82,58
which expressed the form of adding numbers.
Last
96.20
95.80
93.15
81.68
For example, "4+2" in Table means the weight of original labels decays to the wanted threshold in four epochs and remains unchanged during two more epochs. The results shows that as training
Best
96.64
96.32
94.27
81,95
0.65
Last
96.04
95.84
93.42
79.68
Best
96.64
96.24
94.10
79.67
(0.1,
96.12
95.76
91.52
79.37
(0.1.0.8)
Best
95.96
95.28
94.18
85.088
time increases, the accuracy of the last epoch de- creases but not by much. And interestingly, the Best scores get better in some cases because there are more monitored moments.
Last
95.88
94.80
91.52
84.32
Table 8: Results in terms of test accuracy (%) with dif- ferent warm-up times on Trec and Chasenticorp under different noise.
D.3 The trend of correctness statistic
Figure displays the trend of the correctness statis- tic ratio' along with the curve representing the

---

Tree
Trec
Chnsenticorp
Chnsenticorp
1.2
TM-
6975
33
041
6..00
1
0.5
41
0900
6872
10
52
60
13
30
40
59
13
21
36
10
43
52
(a) 20% Sym
(b) 40% Sym
(c) 10% IDN
(d) 40% IDN
Figure 7: The trend of correctness statistic and the proportion of samples correctly partitioned into the clean under different values.
proportion of samples correctly partitioned into the clean under different P2 values during warm-up. Each point on the correctness statistic curve corre- sponds to a monitored situation. We observe the appearance of a turning point in various scenarios, and its occurrence time is determined by the speed at which the model fits the training samples. When the turning point appears, the frequency of occur- rences within fixed range quickly rises. Therefore, set the value of r) to 3 and the corresponding range e 0.01. This is a moderate choice that can accommodate various noise scenarios. A larger is also acceptable but would result a later early stop time.
For P2. we compare two values, 0.5 and 0.2. Un- der simpler noise settings, such as with 20% sym- metric noise, the two P2 curves remain relatively flat, indicating that the cleanliness of the top 20% and top 50% sets is similar. However, under more complex settings, such as with 40% idn, P2 = 0.2 exhibits some advantages. Additionally, a smaller P2 value can accommodate higher noise ratio.

---

Algorithm Adaptive Warm-up
Input: 0 and Ç. training set (X,Y), Y). monitoring interval 8, converge range é, times accuracy
fraction P1
Initial: P= 0, enter False;
while MaxStep do
Draw mini-batch yb); E (I,... B)} from (X,Y)
Optimize and by Equation 1
standard training
/* monitoring the training process at intervals
x/
/ mod 8 = 0 then
Compute rf by Equation 3, (xx, mn) E (X,Y);
m E r
// update the correctness statistic calculate the proportion samples that have been fitted // determine the range that ratio' stays inside
ratio
Σ T( 0/:
P+PP P-P
ratio
if not enter then
if P has same items then
enter True:
// enter the phase // calculate the space available for fur warming up
Compute training accuracy ace: stop_ace= ace + (1 acc) P1
end
else
Compute training accuracy acc; if acc 2 stop_acc then
Break; end
// stop warming
end
end end
Output: 0 and S, MM {mm))
// output trained model and correctness statistic


Reviews:
[{"review_id": "q0ZYaePYl0", "reviewer": "Reviewer_SWmB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the significance of our work. We respond to your comments below:  \n***************************\n(R1). **The experiments and study are limited to classification setting, where most of the datasets are relatively easy. The experimental setting raise some questions (1) If some of assumptions only hold when a learning task is easy? E.g., clean labels are easy to learn compared to incorrect labels. The RTE task and CoLA task are relatively difficult compared to current tasks, does the assumption still hold true?**  \n>1) We focus on classification task because there is an obvious distinction between classification and generation tasks in addressing label noise. Although classifying such datasets without label noise is straightforward for current pre-trained models, achieving accurate classification becomes challenging in the presence of label noise. For example, BERT-FT (fine-tuning noisy data directly) obtains 97.04% on Agnews without noise while only 75.84% with 40% IDN.\n2) Yes, the assumption still hold true in these relatively difficult tasks. We download RTE dataset and corrupt its labels as depicted in our paper to verify it. In the early phase, the label recall rate of clean samples still reaches a higher level than that of erroneous samples, although this trend is somewhat smoother compared to experimental datasets.  \n******************************************************\n(R2). **Given the first assumption, there may be different cases: the clean labels may not be fast to learn compared to incorrect labels consider that the data difficulty level is different. Additionally, considering that annotation errors are more likely to occur in challenging tasks, employing early stopping based on this assumption may lead to the learning process not adequately covering these intricate cases. The label noises are synthetically generated, posing a potential risk of introducing bias that deviates from real-world settings.**  \n>1) Indeed, difficult samples do require more time to learn. However, in the presence of label noise, most clean labels are faster to learn than erroneous ones. Moreover, for the more complex samples that were not learned during the warm-up stage, the subsequent hybrid training stage continues to learn them along with the noisy samples.  \n2) We experiment on the synthetic setup for three reasons: 1) it is easier to explore the effects of various noise scenarios on the model to analyze the aspects of the method that need to be improved; 2) for consistent comparison of previous work ; 3) to the best of our knowledge, there is no benchmark for testing label noise in NLP so far. In the future work, we will continue to test our method in real-world classification scenarios.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The learning with noise is a widely used setting and very important problem.   The paper introduces an adaptive textual label noise learning framework, which is a novel approach to handling unpredictable and mixed label noise in real-world scenarios.  The adaptive warm-up stage with an early stopping method is a key contribution of the paper. It dynamically terminates the warm-up process based on the model's fit to different noise scenarios, ensuring efficient and effective utilization of resources. The authors propose several assumption and provide a discussion ground and form deep insights of noise learning setting. ", "reasons_to_reject": "1. The experiments and study are limited to classification setting, where most of the datasets are relatively easy. The experimental setting raise some questions (1) If some of assumptions only hold when a learning task is easy? E.g., clean labels are easy to learn compared to incorrect labels. The RTE task and CoLA task are relatively difficult compared to current tasks, does the assumption still hold true? \n\n2. Given the first assumption, there may be different cases: the clean labels may not be fast to learn compared to incorrect labels consider that the data difficulty level is different. Additionally, considering that annotation errors are more likely to occur in challenging tasks, employing early stopping based on this assumption may lead to the learning process not adequately covering these intricate cases. The label noises are synthetically generated, posing a potential risk of introducing bias that deviates from real-world settings.\n ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents an adaptive textual label noise learning framework to address the challenge of unpredictable and mixed label noise in real-world scenarios. The framework uses pre-trained models and consists of two stages: an adaptive warm-up stage and a hybrid training stage. The warm-up stage employs an early stopping method based on the training set to dynamically terminate the process according to the model's fit to different noise scenarios. The hybrid training stage incorporates generalization strategies to gradually correct mislabeled instances, effectively utilizing noisy data. Experimental results on multiple datasets demonstrate that the approach outperforms or performs comparably to state-of-the-art methods, even in scenarios with mixed types of noise.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Nen0zT9Hln", "reviewer": "Reviewer_fkV9", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review, here are our response to your commonts:  \n******************************\n(R1). **complex pipeline with numerous ad-hoc parameters and choices**  \n>1) Our approach first warm up the pre-trained model and then train it by incorporating some generalization strategies. Such training mode is adopted in most noise learning methods such as Han et al.(2018), Jindal et al. (2019), Qiao et al. (2022) . \n2) Differently, we introduce some hyper-parameters to extend the applicable types and scenarios, but also reduce other parameters for their specific modules like GMM in selfmix (Qiao et al., 2022) and data selection in co-teaching (Han et al., 2018). Specifically, the parameters for determining the early stopping point based on the change trend of the label recall curves, and the parameters for linear decay fusion are set to cover possible noise range. Due to limited space, the details for setting parameters are omitted. We will attach them in the appendix. Lastly, compared to other approaches such as co-teaching, the number of such parameters falls within an acceptable range. \n \n>References:  \nBo Han, Quanming Yao, et al. 2018. Co-teaching: Robust training of deep neural networks with extremely noisy labels.\nIshan Jindal, Daniel Pressel, et al. 2019. An effective label noise model for DNN text classification.\nDan Qiao, Chenchen Dai, et al. 2022. SelfMix: Robust learning against textual label noise with self-mixup training.\n*************************\n(R2). **some aspects such as mixup do not seem well motivated to address the noise case**  \n>We apologize for not explaining this motivation clearly. In fact, mixup creates more sample points near decision boundaries to make the model capable of dealing with the confused samples in complex noise cases such as IDN. Besides, mixup facilitates mining implicit relations between sentences and learning presentations, which is beneficial to predict accurate pseudo-label as optimize targets. In the ablation study (Sec 5.1), the result of removing the mixup component also shows that it is useful to keep the stability and robustness of the model especially under the 40% IDN. \n**********************\n(R3). **experiments are a synthetic setup with added noise of different types and it is not clear how well the approach will work on real data**  \n>We experiment on the synthetic setup for three reasons: 1) it is easier to explore the effects of various noise scenarios on the model to analyze the aspects of the method that need to be improved; 2) for consistent comparison of previous work ; 3) to the best of our knowledge, there is no benchmark for testing label noise in NLP so far. In the future work, we will continue to test our method in real-world classification scenarios.\n*****************************\n(R4). **many typos and language issues**  \n>Thank you for giving a thorough review. We have proofread the complete paper carefully and tried to correct all the typos and mistakes. E.g. line 409 “its opposite category” is changed to “their opposite category”, line 462 “remains” is changed to “remain”, line 582 “GMM are used… and to divide …” is changed to “GMM is used … and divide” and etc.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ dealing with noise is an important problem\n+ paper is mostly well written", "reasons_to_reject": "- complex pipeline with numerous ad-hoc parameters and choices\n- some aspects such as mixup do not seem well motivated to address the noise case\n- experiments are a synthetic setup with added noise of different types and it is not clear how well the approach will work on real data\n- many typos and language issues", "questions_to_authors": "", "paper_topic_and_main_contributions": "Adaptive Textual Label Noise Learning based on Pre-trained Models\n\nIn this paper the authors describe a pipeline to improve the robustness of learning methods to different types of label noise. They try to identify when a model starts to overfit data due to noise. Based on this they divide the data into \"clean\" and \"noisy\" sets. They then apply pseudo labels to both sets with decaying weights, and also generate novel examples with mixup and encourage accuracy with r-drop. They show experimental results on some datasets using synthetically added noise.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZVeYayo68U", "reviewer": "Reviewer_keZc", "summary": "", "questions": " I will reconsider my score in the rebuttal", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thorough review and constructive advices. We respond to your comments below:  \n**********\n(R1). **The paper does not discuss the computational cost of their approach in detail. Given that it involves a pre-training phase, the computational requirements could be significant.**  \n> Our method doesn’t involve a pre-training phase, but rather fine-tuning the pre-trained model which contains the adaptive warm-up stage and hybrid training stage. Indeed, compared to normal warm-up in other approaches like co-teaching (Han et al., 2018) and selfmix (Qiao et al., 2022), adaptive warm-up needs the extra cost of inferring the whole training set at intervals to update the correctness statistic. To save this cost, we set the interval to 1/10 epoch in our experiments. In addition, in hybrid training stage, we save the cost of inferring the training set and fitting the GMM compared to selfmix. We show the time cost of BERT-FT (fine-tuning noisy data directly), selfmix and our method in the same setiting as example:  \nMethod\\Dataset  |    Trec     |  Chnsenticorp  |   Agnews  \n---|:--:|:---:|---:  \nBERT-FT             |   3min    |       22min    |        2h17min  \nselfmix                  |  5min     |      42min    |        4h40min  \nours                       | 5min      |     43min     |       4h48min  \n \n>References:  \nBo Han, Quanming Yao, et al. 2018. Co-teaching: Robust training of deep neural networks with extremely noisy labels.  \nDan Qiao, Chenchen Dai, et al. 2022. SelfMix: Robust learning against textual label noise with self-mixup training.   \n**********\n(R2). **While the paper claims that the method works well in various noise scenarios, it's not clear if the method is practical for all types of noise. For instance, it might not be effective in scenarios where noise is extreme or highly structured.**  \n>1) We conduct experiments within the noise range of 40% for two reasons: 1) considering that the ratio of corrupted labels in real-world datasets is reported to range from 8.0% to 38.5% (ref1); 2) making consistent comparison with previous work. For extreme scenarios, we conduct experiments with 50%, 60% and 80% label noise on Trec and Chnsenticorp. The performance of baselines and our method drops dramatically as the noise ratio increases. However, our method can still achieve good gains under different types of noises. We show the results on Trec with different level of symmetric noise as example (Best (Last)):\n Method\\Ratio  |    50     |  60  |   80  \n---|:--:|:---:|---:  \nBERT-FT             |   61.6 (48.8)    |      42.8 (28.6)   |       31.6 (20.0)\nselfmix                  |  68.8 (54.4)   |       47.8 (34.2)    |      37.0 (30.8)\nours                       |  69.2 (68.2)    |     48.6 (36.7)    |       37.8 (29.6)\n2) Structured noise may exist in the generation task whose label is a sentence, but the label of classification task in our focus is a category. In fact, IDN is structured noise to some extent whereby noisy labels are predicted by the same model that has the same bias.\n\n>References:  \nSong H, Kim M, et al. 2022. Learning from noisy labels with deep neural networks: A survey.  \n*************************\n(R3). **The paper lacks a deeper analysis of the reasons why their approach performs better than other methods. Some insights into the model's behavior could further enhance the understanding of why their proposed method works well.**  \n>Thank you for the suggestion. We supplement some analysis details in the paper including: 1) the motivation of introducing early stopping; 2) setting different warm-up time for other approaches that contain the warm-up stage; 3) the learning curves during the whole training phase and the curves of the correctness statistic during warm-up; 4) more analysis about the role of the linear decay fusion strategy under different noise settings.  \n*****************************\n(R4). **It's not clear how much the performance of the proposed method depends on the quality of the pre-trained models used. This is an important factor that could influence the applicability of their method in different settings.**  \n>In our experiment setting, we use different models for different language task. They both work well and show the same performance trend, which verifies the generalization of our method on pre-trained models. In addition, we compare the performance based on BERT, RoBERTa and ALBERT on Trec to check the dependency of the proposed method. The result show that: 1) The performance and robustness are all improved using different pre-trained models; 2) The performance of the proposed method does depend on the quality of the pre-trained model and shows a positive correlation. For example, the performance of RoBERTa is comparable to that of BERT on BERT-FT and our method, while ALBERT underperforms BERT by 4.84 points on BERT-FT and 4.054 points on our method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper addresses a very relevant problem in NLP and Machine Learning, i.e., dealing with label noise. The approach is novel and could be very useful in real-world scenarios where label noise is prevalent.\n2. The adaptive warm-up stage with a dynamic early stopping method is a clever idea that prevents overfitting to the noise in the early training process.\n3. The hybrid training stage, which incorporates several generalization strategies, is an effective approach to handle the correction of mislabeled instances.\n4. The experiments are comprehensive, covering a variety of noise scenarios. The results convincingly demonstrate the robustness and effectiveness of the proposed method.", "reasons_to_reject": "1. The paper does not discuss the computational cost of their approach in detail. Given that it involves a pre-training phase, the computational requirements could be significant.\n2. While the paper claims that the method works well in various noise scenarios, it's not clear if the method is practical for all types of noise. For instance, it might not be effective in scenarios where noise is extreme or highly structured.\n3. The paper lacks a deeper analysis of the reasons why their approach performs better than other methods. Some insights into the model's behavior could further enhance the understanding of why their proposed method works well.\n4. It's not clear how much the performance of the proposed method depends on the quality of the pre-trained models used. This is an important factor that could influence the applicability of their method in different settings.", "questions_to_authors": " I will reconsider my score in the rebuttal", "paper_topic_and_main_contributions": "1. Proposing an adaptive textual label noise learning framework that integrates an adaptive warm-up stage and a hybrid training stage. This framework is unique in its utilization of pre-trained models to tackle label noise.\n2. Introducing a dynamic early stopping method for the warm-up process, which assesses the model's fit level to different noise scenarios.\n3. Implementing generalization strategies in the hybrid training stage to correct mislabeled instances, thereby making better use of noisy data.\n4. Demonstrating through experiments that the proposed method performs comparably or even surpasses state-of-the-art methods in various noise scenarios, including those with a mixture of multiple types of noise.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces an adaptive textual label noise learning framework, incorporating both an adaptive warm-up stage and a hybrid training stage that utilizes pre-trained models to address label noise. Additionally, it presents a dynamic early stopping method for the warm-up phase, evaluates the model's adaptability to various noise scenarios, and integrates generalization strategies in the hybrid training stage to optimize the utilization of noisy data. Experimental evidence demonstrates the framework's robust performance across different noise scenarios, including those with a noise range of up to 40% and mixed noise types.

There is a consensus among reviewers regarding the significance of the label noise problem in the fields of NLP and ML. Most reviews emphasize that the experiments demonstrate the robustness and effectiveness of the proposed method, particularly in synthetic noise scenarios. However, some concerns center around the simplicity of the tasks and the nonapparent computational complexity analysis of the method. It would be beneficial for the authors to offer further clarification regarding these computational complexities and the potential challenges stemming from the multitude of ad-hoc parameters and choices in future versions of the paper. Addressing these points would enhance the paper's applicability to real-world scenarios where label noise is prevalent and facilitate future iterations of this research.
Decision: Accept-Findings