Addressing NER Annotation Noises with Uncertainty-Guided Tree-Structured CRFs
Authors: Jian Liu
Weichang Liu
Yufeng Chen
Jinan Xu
Zhe Zhao
Venue: EMNLP 2023 Main (2023)
Keywords: Named Entity Recognition (NER)
Partial and Incorrect Annotation
Uncertainty
constituency tree parsing

Abstract: Real-world named entity recognition (NER) datasets are notorious for their noisy nature, attributed to annotation errors, inconsistencies, and subjective interpretations. Such noises present a substantial challenge for traditional supervised learning methods.
In this paper, we present a new and unified approach to tackle annotation noises for NER. Our method considers NER as a constituency tree parsing problem, utilizing a tree-structured Conditional Random Fields (CRFs) with uncertainty evaluation for integration.
Through extensive experiments conducted on four real-world datasets, we demonstrate the effectiveness of our model in addressing both partial and incorrect annotation errors. Remarkably, our model exhibits superb performance even in extreme scenarios with 90\% annotation noise.

Noisy NER with Uncertainty-Cuided Tree-Structured CRFs
Jian Liu'; Weichang Liu'; Yufeng Chen', Jinan Xu', Zhe Zhao²
Beijing Key Lab of Traffic Data Analysis and Mining, Beijing Jiaotong University Tencent Al Lab
(jianliu, weichangliu, chenyf, jaxu) @bjtu. nlpzhezhao@tenneet. com
Abstract
51:
Newcobe
chose the Los Angeles Lakers
Label:
O B-LOC I-LOC
Real-world named entity recognition (NER) datasets are notorious for their noisy nature, at- tributed to annotation errors, inconsistencies, and subjective interpretations. Such noises present a substantial challenge for traditional supervised learning methods. this paper. we present new and unified approach to tackle annotation noises for NER. Our method considers NER as constituency ing problem, utilizing tree-structured Con- ditional Random Fields (CRFs) with uncer- tainty evaluation for integration. Through ex- tensive experiments conducted on four real- world datasets, we demonstrate the effective- ness of model in addressing both partial and incorrect annotation erros. Remarkably, our model exhibits superb performance even in extreme scenarios with 90% annotation noise.
B-PER
B-ORG I-ORG 1-ORG
Partial Annotation
Incorrect Annotation
Figure I: Two typical annotation noises, taken from the CoNLL 2003 NER benchmark.
partial annotation noise, previous works have em- ployed re-weighting mechanisms to mitigate the impact of false negatives (Peng et al., 2019; Jie et al., 2019). For the incorrect annotation noise, previous works have suggested that the of rein- forcement learning and teacher-student framework effective (Zhang al., 2021; Zhou al., 2022). Nevertheless, most works focus on only one type of noise, and there has been limited success using a unified approach to handle diverse types of noises (Qu et al., 2022; Huang et al., 2021b). Additionally, due to the combination of different components, previous methods may suffer from error propagation (Peng et al., 2019; Wang al., 2019b).
pars-
1 Introduction
Named entity recognition (NER) a fundamental natural language processing task that aims to find entities with certain types in texts (Ma and Hovy, 2016; Lample et al., 2016). Generally, building high-performance NER model requires obtain high-quality labeled data. However, due the complexities of the labeling process, real-world NER datasets often contain annotation noises (Lan et al., 2020; Huang et al., 2021a), For example, Figure 1 gives two typical annotation errors in the CoNLL03 benchmark (Tjong Kim Sang and De Meulder, 2003) the PERSON entity "New- combe" is missed by annotators, and "Loc An- geles Lakers" incorrectly labelled LOCA- TION. Indeed, the noise rate is high (2%-8%) in real-world datasets (Liu and Tao, 2016; Song et al., 2022), and this issue can significantly im- pede model learning.
In this paper, we present a unified approach for addressing annotation noises in NER. Specif- ically, shown in Figure 2, our method struc- tures NER as constituency tree parsing prob- lem, with each node's label representing the en- tity type of textual span. To learn with noises, we introduce uncertain nodes in the tree, indicat- ing that the annotation for a span may be unreli- able (this is compatible with both partial and incor- rect annotation noises), and we use tree-structured conditional random fields (CRFs) (Rush, 2020) to average such uncertainties for learning, relieving the model from over-training on the noises. We present an Monte Carlo Dropout (MC-Dropout) mechanism (Gal and Ghahramani, 2016) for eval- uating uncertainty and demonstrate that the over- all framework compatible with an iterative co- learning framework.
To date, many studies have been proposed to ad- dress NER annotation noises. For example, for the
"Equal contribution.

---

To verify the effectiveness of our method, have conducted extensive experiments on real- world and simulated datasets. According to the results, on real-world datasets including Youku (Yang et al., 2020) and Weibo (Peng and Dredze, 2017), our approach outperforms previous state- of-the-art methods by up to 3.27% in absolute Fl score (6 5.1), demonstrating remarkable effective- ness in addressing both partial and incorrect an- notations. On simulated data sets, our method ex- hibits advantages under extreme conditions. For example, in the case 90% partial or incorrect an- notations on the TaoBao dataset (Jie et al., 2019), our approach achieves an impressive Fl score of 77.3% outperforming previous state-of- the-art methods by a margin of 10.4% ($ 5.2 and 5.3).
unceraiity
ROOT
DRG
ORG
PER
Newcobe
chose
the
Los
Angeles
Lakers
Figure 2: streestructured CRFs with uncertain/latent
node. The uncertainty node means that the annotated label may not be reliable.
(Li et al., 2020, 2022). Regarding incorrect anno- tation noise, prior approaches have primarily fo- cused on reinforcement learning methods (Yang al., 2018; Luo al., 2020; Chen et al., 2020), teacher-student frameworks (Wang et al., 2019b; Liang et al., 2020), and advanced noise filtering mechanisms (Zhang et al., 2021; Qu et al., 2022; Wang et al., 2023; Huang et al., 2021b). However, these methods are often tailored to address spe- cific types of annotation noise, limiting their abil- ity to generalize to different scenarios. In contrast, research aims propose unified framework capable of effectively handling various types of annotation noises simultaneously, yielding more comprehensive and versatile solution.
In summary, our contributions are follows:
We have introduced new framework that can effectively handle NER annotation noises. This framework stands out for its uni- fied approach in addressing partial and incor- rect annotations.
We have introduced tree-structured CRFs with uncertainty nodes for learning, along with the incorporation of MC-Dropout for uncertainty evaluation. This approach pro- vides a fresh perspective on noise modeling that can be applied to other tasks.
We have conducted extensive experiments on both real-world and simulated datasets, demonstrating promising results. To promote further exploration, we have made our code publicly available at https: / com/feili583/we
2.2 NER as Parsing
Our work also relates to research that takes a struc- tured prediction perspective, particularly consti- tutional parsing, on NER (Finkel and Manning, 2009; Yu et al., 2020; Fu et al., 2021a; Gómez- Rodríguez and Vilares, 2018). For example, Wang et al. (2019a) improve Chinese NER by incorpo- rating features from neural semi-CRFs and neu- ral tree-CRFs. Fu et al. (2021b) apply parsing techniques to handle nested NER and introduce latent nodes for likelihood integration. Yang and Tu (2022) demonstrate the ability to help entity boundary detection by leveraging structure fea- tures. Motivated by these prior approaches, we structure NER as constituency tree prediction problem. However, extend this perspective by introducing uncertain nodes effectively handle annotation noises. We hope that this new approach to addressing noises has the potential to inspire further research other tasks.
2
Related Work
2.1 NER with Annotation Noises
The issue of addressing annotation noises in NER has received significant attention the research community (Yang et al., 2018; Jie et al., 2019; Li et al.,, 2022). In the case of partial annotations, pre- vious methods have explored various techniques, including the utilization of partial CRFs (Yang et al., 2018 8; Jie et al., 2019), binary classification approaches (Shang al., 2018), and PU learn- ing strategies (Mayhew al.. 2019; Peng et al., 2019). Recent works have introduced innovative strategies such as adaptive negative sampling to ef- fectively prevent the inclusion of unlabeled noises

---

Newcobe
chose
the
Los
Angeles
Lakers
Uncertaiity-Guiiee Tree CRFs for Leaminn
Uncertainty Evaluation via MC-Dropout
Uncertainty
(Newcobe)
(Los
O
Angeles Lakers)
LOC
PER
PER
ORG
ORG
Newcobe
chose
the
Los
Angeles
Lakers
Span Uncertainties
Uncortainty Re Evoluation
Figure 3: Overview of our approach. Specifically, we treat NER as a constituency tree parsing problem and utilize uree CRFs with uncertain nodes to handle noisy annotations ( 3.1). The MC-Dropout is used uncertainty evaluation 3.2), and the above two modules form iterative learning framework to improve each other (g 3.3).
3 Approach
mulated follows (Dozat and Manning, 2017):
Our approach, depicted in Figure 3, consists of three main components:
score(T, Σ 2.j.K i.j.k Suj.
(1)
and the (logarithm) Gibbs distribution of T over all compatible constituency trees is:
Uncertainty-guided tree CRFs. This module frames NER a constituency tree parsing problem and utilizes tree conditional random fields (CRFs) with uncertain nodes to handle noisy annotations (5 3.1).
log (T score(T,, log Z
(2)
Here Z is normalization term that sums scores of all constituency trees: Z -Erer exp expscore sT)r and in the binary constituency tree setting, we can compute it effectively using the Inside algorithm over a tree CRF's formulation (Lisner, 2016), de- noted by = INSIDE(S). In our approach, we compute each entry in the score tensor S using bi- affine attentions (Dozat and Manning, 2017):
MC-Dropout for uncertainty evaluation. This module estimates the labeling uncertainty for text span using MC-Dropout (Gal and Ghahramani, 2016), and integrates it into tree CRFs for learning with noises ($ 3.2).
Iterative co-learning mechanism. This mech- anism enables interactions between the above two modules, allowing them to learn from each other for improvement ($ 3.3).
Sij.k = H.W.( H,+ (H, H2) W (2)
(3)
where H; Id is the representation of word x from a BERT encoder (Devlin al., 2019), and w("). wr(") C IRdxd are parameters optimized through the training process. When we have ac- to the ground-truth constituency tree, denoted as Tx, we can directly maximize logx(TXxA> (X) for learning. For inference, given S, we can apply the Cocke-Younger-Kasaml (CKY) algo- rithm (Sakai, 1961) to find the most probable con- stituency tree with an O(N3) time complexity.
3.1 Uncertainty-Guided Tree CRFs
The uncertainty-guided tree CRFs covert NER as constituency tree parsing and incorporate uncer- tainty mechanisms to address annotation noises. Let consider sentence X with N words: X #2,,,2,,., .NN) A constituency tree for the sen- tence can be represented as a rank-3 binary tensor E N where the entry T.j.k = indi- cates that the span [.;...., [.]] assigned an en- tity label & from a label set IK (denoted as [i, jl >> k). Let us assume that we can acquire a score tensor S E IR N where S.j.k represents the log potential for assignment li, j] + k. Then the score! of the constituency tree T can be for-
Nevertheless, the above formulation is applica- ble only when the clean constituency tree Tx is available. In our problem, are confronted with noisy labels, which could result in mis-training. To address this challenge, we introduce uncer- tain nodes associated with the tree, represented by U E N Here, U.j,k indicates the uncer- tainty (or risk) of the model assigning label & to the
We assume l-order representation.

---

span Lú, j] during the learning process. Different from previous methods learning transition prob- abilities from correct labels to nosiy ones (Wang et al., 2019b), we propose a more principled ap- proach by integrating uncertainties over all com- patible constituency tree, and obtain an average score for sentence X:
Newcobe chose the Los Angeles Lakers
caaadidtte spars
(Newcobe) (chose) (the)
O
MC-cropout
PER
LOC
PER
PER
PER LOC
(Newcobe chose) (chose the)
ORG
prediction set
score U) = log E exp(
EU
Figure The process of uncertainty evaluation, using the first I-word span "Newcobe" for illustration.
(4)
Notice that the noisy labels are not used to com- pute this score, and in the sense that we integrate all constituency tree out, it resembles unsuper- vised learning (The noisy labels are used mea- sure the uncertainties). For learning, maxi- mize:
where W E k are model parameters.
It might be alluring to directly oij for con- structing the uncertainty tensor, but according previous research, the output of a network is of- ten poorly calibrated, which can output very high probability even when the results is not reliable (Guo et al., 2017). Here motivated by Gal and Ghahramani (2016), we introduce MC-Dropout to evaluate the uncertainty. Specifically, after train- ing the model using the original labeled data, we use it to predict the labels for T times but with dropout layers activated. In this way, for span, we can obtain prediction set:
logp(S, = scorc(S,U) log Z (5)
It should also note that directly computing score(S, U) is infeasible due to the exponential many terms. We show that has a connection with the original inside algorithm by absorbing the un- certainty tensor into the score tensor and thus use the original Inside algorithm:
[i.l [y), y2..... yTl
(10)
score( U) INSIDE(Log U @ S)
(6)
According Gal and Ghahramani (2016), the variance of Y(.J] indicates the uncertainty of the prediction. We therefore formulate as cate- gorical distribution, and then for each entity label K E K, the entity probability parameter is:
where is element-wise add operator. Notice that the above process only applies at the training and does not affect the inference procedure.
3.2 Uncertainty Evaluation via MC-Dropout
exp (k)
This modules aims to measure the uncertainty ten- sor U using the original labels. Specifically, for an input sentence X, we first employ another BERT (Devlin et al., 2019) encoder to learn the represen- tations H E Nxd.
Gij.A
Σ exp ((*)
(11)
with (Kk being the frequency of seeing in the prediction set. Figure 4 showcases running ex- ample for uncertainty estimation.
H Encoder(X)
(7)
3.3 An Iterative Co-Learning Mechanism
We utilize an iterative co-leaming mechanism where tree CRFs and uncertainty evaluation mod- ules alternate as teacher and student roles.
Then for each span [i.j], we build representation Raj E IR following Li al. (2020):
Step 1: Initial Uncertainty Estimation. We es- timate the initial uncertainty tensor U by training on the noisy labels from the training set, capturing the model's confidence in its predictions. Step 2: Training the Parsing Model. The pars- ing model trained by integrating the estimated uncertainties, adapting its learning process based on varying confidence levels associated with la- beled data.
Raj H, H + H, H2) e H Hj)
where and are clement-wist plus and mul- tiplication operators. We then build classifier to compute the probability for span being each entity label k K:
oi = softmax W R.g)
(9)

---

Step 3: Uncertainty Re-Evaluation. We refine uncertainty estimation by retraining the model us- ing prediction results from the parsing model, en- hancing its ability to estimate uncertainties effec- tively. We then alternate between Step 2 and Step 3 until convergence.
4.2 Baselines for Comparison
We compare our approach with several state-of- the-art methods, including: BERT, which utilizes the original BERT model for NER, without con- sidering any specific mechanism to address an- notation noises. Weighted PA (Jie al., 2019), which incorporates marginal probability to handle incomplete annotations. To ensure fair compar- ison, we modify it by integrating a BERT repre- sentation. NegSampling (Li et al., 2020), which adopt negative sampling to relieve the problem of over-training on false negatives. Con-MPU (Zhou et al., 2022), which employs PU learning to handle partial annotations. BOND (Liang et al., 2020), which adapts self-training techniques to tackle dis- tant supervision scenarios. NNCE (Liu et al., 2021), which identifies noisy annotations by us- ing confidence thresholds. We also incorporate modifications to BERT representations for com- parison purposes. SCDL (Zhang et al., 2021), which utilizes a teacher-student framework to se- lect pseudo-labels. ATSEN (Qu et al., 2022), which augments the teacher model with an adap- tive mechanism and student ensemble. Partial Tree (Fu et al., 2021b), which utilizes partially ob- served Tree CRFs address nested NER.
4 Experimental Setups
4.1 Datasets and Evaluations
We conduct experiments on four NER datasets: Youku (Yang al., 2020), Weibo (Peng and Dredze, 2015, 2017), CoNLL-2003 (Wang et al., 2019b), and TaoBao (Jie et al., 2019). Our real- world evaluation settings are as follows: 1) For the Youku dataset, we directly adopt it for train- ing and testing, which consists of noisy training set with entity annotations and clean test set. 2) For the Weibo dataset, we the original noisy version for training and validation based on (Peng and Dredze, 2015). For testing, we utilize the corrected version mentioned in (Peng and Dredze, 2017). 3) In the case of the CoNLLO3 dataset, we use the original training and validation sets from (Tjong Kim Sang and De Meulder, 2003). Addi- tionally, we incorporate the corrected test set pro- vided in (Wang al., 2019b) for evaluation. In addition, we perform simulated experiments to assess the performance of our model under sep- arate cases: partial annotations and incorrect an- notations, well as extreme conditions. Re- garding the Weibo dataset, divide the noisy training into three parts?, using the corrected version (Peng and Dredze, 2017) as reference: clean set Sclean partially annotated set Spartial (ac- counting for 27.9%), and incorrectly annotated set Sincorrect (accounting for 8.49%). We com- bine Selean and Spartiaa to form dataset contain- ing only partial annotation noise, and we combine Selean and Sincorrect to form a dataset containing only incorrect annotation noise. For the CoNLLO3 and TaoBao datasets, which exhibit lower levels of noise, we adopt the following strategies con- struct evaluation datasets: 1) Partial Annotation: randomly masking labels for a certain percentage of entities, and 2) Incorrect Annotation: randomly substituting certain percentage of entity annota- tions with labels from other types.
4.3 Implementations
In our implementations, we use BERT base encoder (Devlin et al., 2019) as the basic feature extractor. For Tree CRFs, the learning rates on the Youku, Weibo, and CoNLL-2003 datasets are set to le- 5, while the TaoBao dataset, it is set to 3c-5. These values chosen from the options: 1e-5, 5e-5 and le44. The batch size is set to 48, and we train with a window context 64 words. For the uncertainty evaluation step. we employ MC- Dropout with a different number: We set T=30 for the Youku dataset and T=10 for the other datasets. These values are determined through grid search process. To control the training process, we set maximum of 20 epochs and use a time decay learning rate strategy. This strategy ensures that the training terminates after reaching the maxi- mum number of epochs.
Experimental Results
5.1 Evaluations Real-World Datasets
We first conducted an evaluation of our method on NER datasets that include natural an- notation errors introduced by annotators (such as
In the cases where a sentence contains two types of er- rors, we correct the partial annotations and categorize them as incorrect annotations.

---

Youku
Weibo
CoNLL03
Approach
P
R
F1
P
R
F1
R
FI
BERT
73.81
29.71
42.37
52.45
36,06
42.74
90.54
91.14
90.84
Partial Annotation Settings
Weighted- PA (Jie al., 2019)
71.62
74.72
73.14
57.30
52.71
85.58
87.73
86.65
NegSampling et al., 2020)
68.10
68.59
68.35
64.15
$5.31
59.40
93.23
92.23
92.73
Conf-MPU (Zhou al., 2022)
61.77
64.69
58.78
56.77
36.47
44.41
80.79
89.55
84.94
Incerrect Annolation Scitingss
BOND (Liang al., 2020)
70.64
33.61
45.55
59.24
49.03
53.65
92.00
92.63
92.31
NNCE (Liu et al., 2021)
71.48
74.21
72.82
49.02
59.81
53.88
80.99
85,40
83.14
SCDL (Zhang al., 2021)
70.36
48.69
57.55
62.16
35.20
44.95
93.25
91.94
92.59
ATSEN (Qu et al., 2022)
71.47
52.74
60.69
59.36
38.01
46.35
93.32
92.27
92.79
Nested Enfily Sclling
Partial-Tree (Fu al., 2021b)
69.73
66,68
68.17
64.14
54.39
58.86
92.24
92.51
92.37
Ours
75.72
77.11
76.41
61.10
59.18
60.12
93.47
92.16
92.81
Table I: Results on three real-world NER benchmarks, where the previous methods are categorized by the scenar- ios they were originally proposed for.
Approach
P
F1
on CoNLL03 (%)
on TaoBao (%)
100
BERT
57.53
40.19
47.32
Weighted (2019)
65.92
56.46
60.82
80
NegSampling (2020)
66.63
57.87
61.91
Conf-MPU (2022)
24.18
15.94
19.21
BOND (2020)
64.98
50.00
56.52
ours
aura
BERT
BERT
NNCE (2021)
49.13
60.53
54.23
Waighttd PA
Weighted PA
SCDL (2021)
54.81
29.01
37.94
40
NegSampling
NegSamoling Conf-MPU
ATSEN (2022)
57.26
36.22
44.38
Conf-MPU
Partial- Tree (2021b)
72.52
47.08
57.09
NNCE
NNCE
SCDC
Ours
65.97
60.87
63.32
ATSEN
ATSEN Partsi Tree
Partall Tree
Table 2: Results on Weibo dataset with partial annota- tion settings.
0.2 0.4
0.8 Partial annotation ratio
0.6
0.2
0.4
0.6
0.8
Partial annotation ratio
Results on CoNLL03 and TaoBao datasets regarding different partial annotation ratio. The perfor- mance is based on a 5-run average.
Figure 5:
Weibo and CoNLL-2003) machines (such as Youku). As illustrated in Table 1, our method outperforms all other approaches, achieving the highest performance. Notably, when compared to the traditional BERT model, our method exhibits substantial improvements in Fl score: 34.04%
in mitigating false negatives partial annotations.
5.2 Evaluations on Partial Annotations
for Youku, 17.38% for Weibo, and 1.97% for CoNLLO3, These results demonstrate the effec- tiveness of our method in practical use.
To gain comprehensive understanding of method's capability in handling partial annota- tions, we conducted more detailed experiments us- ing simulated settings. Table 2 presents the results of our method on the Weibo dataset, where the training set consists solely of clean and partial an- notations (5 4.1). I). The results indicate notable im- provement of 1.41% in Fl score compared to the state-of ofiee-of-th-anne art approach. Moreover, upon closer observed that the improvement primarily stems from an improved recall.
Moreover, our method outperforms strong base- lines in handling both incomplete and incorrect annotations, demonstrating its ability to simulta- neously resolve both categories of errors. It also exhibits improved applicability to datasets with higher level of noises such Youku and Weibo, resulting in an absolute Fl score enhancement of addition, our method obtains
examination,
up to 3.27%. high recall rate, ranking first for Youku and sec- ond for Weibo, which demonstrates its robustness
Figure 5 shows results on Taobao and CoNLLO3 with partial annotation settings (by masking a por-

---

Approach
Approach
R
F1
BERT
52.25
$5.50
53.83
Partial-Tree (2021b)
64.14
54.39
58.86
Softmax Probability
62.30
56.28
59.14
Weighted-PA (2019)
62,38
46.41
53.22
One Hot Prediction
60.36
59.18
59.67
NegSampling (2020)
57.46
63.29
60.23
Conf-MPU (2022)
21.28
16.73
18.73
Ours (MC-Dropout)
61.10
59.18
60.12
BOND (2020)
59.01
66.75
62.64
NNCE (2021)
47.04
43.78
45,35
Table 4: Ablations uncertainty evaluation.
SCDL (2021)
53.95
39.95
45.91
ATSEN (2022)
57.53
43.88
49.78
Partial- (2021h)
61.66
58.77
60.18
egy in scenarios involving incorrect annotations. Figure 6 illustrates results for CoNLL 2003 and Taobao by substituting percentage of entity la- bels with labels from other categories. Accord- ing to the results, our method performs well in the vast majority of cases (especially on the TaoBao datasets), suggesting its effectiveness for resist- ing incorrect annotation noises. The generally de- graded performance on CoNLL 2003 may be due to the dataset's limited entity coverage, and our stimulation strategy may result in numerous in- consistencies. In addition, demonstrate that, compared to partial annotation, the advantage of our method diminishes more challenging cases with higher incorrect annotation ratios (over 70% in CoNLLO3 and 90% in TaoBao). Nevertheless, these extreme circumstances (such as case that 90% annotations are wrong) are less common in real-world scenarios and require further study in future research.
Ours
63.94
61.70
62.80
Table 3: Results Weibo dataset with incorrect anno- tation settings.
F1 on CoNLL03 (%)
F1 TaoBao (%)
80
80
curs
ours
BERT
BERT
Walgrisd PA
Weighted PA
40
NegSampling
NegSampiin
Cont-MPU MPU
CPn
BOND
BOND
NNCE
NNCE
20
SCDL
SCDL
ATSEN
ATSEN
Parial Tree
Partial Tree
0.4 0.6
0.8 Incorrect annotation ratio
0.2
0.4
Incorrect annotation ratio
Figure 6: Results on CoNLLO3 and TaoBao datasets regarding different incorrect annotation ratio. The per- formance is based on 5-run average.
Discussion
To further investigate the effectiveness of our ap- proach, we conduct more in-depth studies, divided into ablation study and case study.
tion of entities), which demonstrate notable mar- gin between our approach and other methods. Re- markably, even when 90% of entities are masked, our method achieves an Fl score of over 80%, outperforming other methods significantly. This highlights the robustness of our approach in resist- ing partial annotation noises.
6.1 Ablation Study
Effects of Uncertainty Evaluation. Using the Weibo dataset as example. we investigate the effects of uncertainty evaluation and compare our approach Partial-Tree (Fu al., 2021b). which treats O-label as latent nodes and does not adopt uncertainty evaluation, Softmax Probabil- ity, which uses the distribution as the uncertainty tensor, and One-Hot Prediction, which uses the argmax operation and constructs one-hot uncer- tainty vector for textual span. Based on the results in Table 4, the methods employing certainty evaluation outperform Partial-Tree, indi- cating that uncertainty evaluation is important in the noisy scenarios. Our method outperforms all others, suggesting that MC-Dropout the most appropriate for evaluating uncertainty. One ex-
5.3 Evaluations on Incorrect Annotations
addition, we conducted experiments utilizing simulate settings to verify the effectiveness of our method for addressing incorrect annotations. Ta- ble shows results on Weibo, where the train- ing set consists of only clean and incorrect an- notations ($ 4.1). According to the results, our method outperforms or behaves similarly to earlier approaches. suggesting its effectiveness. Interest- ingly, we observed the highest precision in this in- correct annotation scenario, indicating that our ap- proach automatically adopts a more cautious strat-

---

Approach
Case 1: Youku Total Entertainment Jin Shijia's documentary
One-War Pass
62.29
54.24
59.23
Gold
B-TY
I-TV
I-TV
BERT-BERT Co-I.eaming
59.64
56.76
58.17
1.0
Tree-Tree Co-Learnig
65.74
48.25
55.65
0.5
Ours (BERT-Tice Co-I.cam.)
61.10
59.18
60.12
1.0
2.0
3.0
4.0
5.0
Table 5: Ablations iterative co- co-learning mechanism.
Iteraiions
Case Leggend of Immortal Sword and Strange Heroes 3
Weibo (%)
Label Gold
B-TV B-TV
LTV LTV
LTV LTV
LTV HTV
0
ITV
HTV
I-V
1.0
0.5
0.0
Itorations
NegSampling
EasterTise
Figure 8: Two cases in the Youku dataset. It empha- sizes progression of uncertainty evaluation in the gold annotation the number of iterations increases. A higher value sigmifies a heightened confidence in the accuracy annotation. The model exclusively re- lies the corresponding annotation the gold label.
BERT
20
Weightad
NNCC
10
BOND
ATSEN
SCOL
Noise ratia
Figure 7: Effects of learning from pure noises.
method consistently outperforms others and shows continuous improvement as more noisy data is added. This indicates that the inclusion of pure noises is beneficial for learning, highlighting the advantages of our integrated training objective. In contrast, certain methods like BERT, BOND, and Weighted PA methods display declining trend, implying that the addition of pure noises has a detrimental impact, leading to negative effects.
planation for the poor performance of Softmax Probability and One-Hot Prediction is that Soft- max Probability's output is typically poorly cal- ibrated and cannot accurately reflect uncertainty (Guo et al., 2017), whereas One-Hot Prediction is prone to error propagation.
Effects of Iterative Co-Learning Mechanism. To investigate the effects of the iterative co- learning mechanism, we compare our method to the following alternatives: One One-Way Pass, which evaluates uncertainties only once and does not use iterative lcarning; BERT-BERT Co-Learning and Tree-Tree Co-Learning, which employ two BERT- based or tree-based models for uncertainty evalu- ation and NER learning. According to Table 5, One-Wir Way Pass yields relatively bad results, indi- cating the significance of iterative co-learning. In addition, the results demonstrate that the BERT and tree CRFs have a complementary effect on un- certainty evaluation and training, and a mismatch will result in degraded performance.
6.2 Case Study
In Figure 8, we present the evolution of uncer- tainty evaluation, represented by the span confi- dence of the ground-truth label, for two examples from the Youku dataset. Through the training pro- cess, our method significantly improves in accu- rately assigning correct labels to the initially mis- labeled entities. Moreover, the model's 's predic- tion confidence consistently increases over time, demonstrating its effective learning from noisy la- bels rather than relying solely annotations.
Conclusion
This study introduces novel and unified solu- tion to effectively address the noisy annotations in NER. Our model integrates tree CRFs with uncer- tainty evaluation, and comprehensive experiments validate the effectiveness of our approach in han- dling partial and incorrect annotations. Although our primary focus was NER, we anticipate that our method holds promise for broader applications
Effects of Learning from Pure Noises. We conduct an interesting exploration to determine if model can learn from pure noises by separat- ing all annotation errors (ineluding partial and in- correct annotations) from the Weibo training set and progressively add them into clean datasets for learning. As demonstrated in Figure 7, our

---

various natural language processing tasks involv- ing annotation noises.
Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. Open- Review.net.
Acknowledgement
Jason Fisner. 2016.
Inside-outside and forward- backward algorithms are just backprop (tutorial pa- per). In Proceedings of the Workshop on Structured Prediction for NLP, pages 1-17, Austin, TX. Asso- ciation for Computational Linguistics.
This work was supported by the Fundamen- tal Research Funds for the Central Universities 2023/BMC058. It was also supported by the National Natural Science Foundation of China (No.62106016 and No.61976016), and the Open Projects Program of the State Key Laboratory of Multimodal Artificial Intelligence Systems.
Jenny Rose Finkel and Christopher D. Manning. 2009. Joint parsing and named entity recognition. In Pro- ceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Compulational Lin- guistics, pages 326-334, Boulder, Colorado. Asso- ciation for Computational Linguistics.
8 Limitations
The study's findings should be considered in light of certain limitations. Firstly, it is important to recognize that our assumption of a domein-free formulation of noises may not fully account for domain-specifie annotation errors. Indeed, real world annotation errors can vary across datasets and domains, emphasizing the need for future in- vestigations that address domain-specific consid- erations. Secondly, a more promising approach to addressing these noises involves modeling the an- notation process itself, rather than relying solely on learning from the noises. One potential solu- tion is to develop an annotation model for each individual annotator to incorporate his/her anno- tation bias, despite the additional costs involved. In summary, exploring these complex issues and their potential applications holds promise for fu- ture research and requires further investigation.
Jinlan Fu, Xuanjing Huang, and Pengfei Liu. 2021a. SpanNER: Named entity re-frecognition as span pre- diction. In Proceedings of the 59th Anmual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Nat- ural Language P Processing (Volume I: Long Papers), pages 11935 Online. Association for Compu- tational Linguistics.
Yao Fu, Chuanqi Tan, Mosha Chen, Songfang Huang, and Fei Huang. 2021b. Nested named entity recog- nition with partially-observed treecrfs. In Proceed- ings of the AAAI Conference on Artificial Intelli- gence, volume 35, pages 12839-12847.
Yarin Gal and Zoubin Ghahramani. 2016. Dropout bayesian approximation: Representing model un- certainty in deep learning. In Proceedings of The 33rd International Conference Machine Learn- ing, volume 48 of Proceedings of Machine Learning Research, pages 1050-1059, New York, New York, USA. PMLR.
References
Carlos Kömez-Rodriguee and David Vilares. 2018. Constituent parsing as sequence labeling. In Pro- ceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pages 1314- 1324, Brussels, Belgium. Association for Computa- tional Linguistics.
Daoyuan Chen, Yaliang Li, Kai Lei, and Ying Shen. 2020. Relabel the noise: Joint extraction of entities and relations via cooperative multiagents. In Pro- ceedings of the 58th Anmual Meeting of the Asso- ciation for Computational Linguistics, pages 5940- 5950, Online. Association for Computational Lin- guistics.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Wein- berger. 2017. On calibration of modern neural net- works. Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1321-1330. PMLR
Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techmologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Associ- ation for Computational Linguistics.
Quzhe Huang. Shengqi Zhu, Yansong Feng, and Dongyan Zhao. 2021a. Exploring distamtly-lbblle rationales in neural network models. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1th Interna- tional Joint Conference cessing ( (Volume I: Long Papers), pages 5571-5582, Online. Association for Computational Linguistics.
Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency pars- ing. In 5th International Conference on Learning
Natural Language Pro-

---

Xiusheng Huang, Yubo Chen, Shun Wu, Jun Zhao, Yuantao Xie, and Weijian Sun. 2021b. Named en- tity recognition via noise aware training mechanism with data filter. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4791 Online. Association for Compu- tational Linguistics.
Tongliang Liu and Dacheng Tao. 2016. Classification with noisy labels by importance reweighting IEEE Transactions on Pattern Analysis and Machine In- relligence, 38(3)4447-411.
Ying Luo, Hai Zhao, and Junlang Zhan. 2020. Named entity recognition only from word embeddings. In Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pages 8995- 9005, Online. Association for Computational Linguistics.
Zhanming Jie, Pengjun Xie, Wei Lu, Ruixue Ding, and Linlin Li. 2019. Better modeling of incomplete an- notations for named entity recognition. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies. Volume (Long and Short Papers), pages 729-734, Min- ncapolis, Minnesota. Association for Computational Linguistics.
Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNS- CRE. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Pupers), pages 1064-1074, Berlin, Germany. Association for Computational Linguis- tics.
Guillaume Lample, Miguel Ballesteros, Sandeep Sub- ramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 260, San Diego, California. Association for Computational Linguistics.
Stephen Mayhew, Snigdha Chaturvedi, Chen-Tss Tsai, and Dan Roth. 2019. Named entity recognition with partially annotated training data. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNT.L), pages 645-655, Hong Kong. China. Association for Computational Lin- guistics.
Ouyu Lan, Xiao Huang, Bill Yuchen Lin, He Jiang, Liyuan Liu, and Xiang Ren. 2020. Learning to con- textually aggregate multi-source supervision for se- quence labeling. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 2134-2146, Online. Association for Computational Linguistics.
Minlong Peng, Xiaoyu Xing, Qi Zhang, Jinlan Fu, and Xuanjing Huang. 2019. Distantly supervised named entity recognition using positive-unlabele learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2409-2419, Florence, Italy. Association for Compu- tational Linguistics.
Yangming Li, Lemao Liu, and Shuming Shi. 2020. Empirical analysis of unlabeled entity problem named entity recognition. arXiv preprint arXiv:0012.05226.
Nanyun Peng and Mark Dredze. 2015. Named entity recognition for Chinese social media with jointly trained embeddings. In Proceedings of the 2015 Conference Empirical Methods in Natural Lan- guage Processing, pages 548-554, Lishon, Portugal. Association for Computational Linguistics.
Yangming Li, Lemao Liu, and Shuming Shi. 2022. Re- thinking negative sampling for handling missing tity annotations. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7188- 7197, Dublin, Ireland. Association for Computa- tional Linguistics.
Nanyun Peng and Mark Dredze. 2017. Supplementary results for named entity recognition on chinese so- cial media with an updated dataset. Technical re- port, Technical report.
Chen Liang, Yue Yu, Haoming Jiang, Siawpeng Tr, Ruijia Wang, Tuo Zhao, and Chao Zhang. 2020. Bond: Bert-assisted open-domain named entity recognition with distant supervision. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '20, page 1054-1064, New York, NY, USA. Association for Computing Machinery.
Xiaoye Qu, Jun Zeng, Daizong Liu, Zhefeng Wang, Baoxing Huai, and Pan Zhou. 2022. Distantly- supervised named entity recognition with adaptive teacher learning and fine-grained student ensemble. arXiv preprint arXiv:2212.06522.
Alexander Rush. 2020. Torch-struct: Deep structured prediction library. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 335- 342, Online. Association for Computational Lin- guistics.
Kun Liu, Yao Fu, Chuanqi Tan, Mosha Chen, Ningyu Zhang, Songfang Huang, and Sheng Gao. 2021. Nobse-labeled NER with confidence estimation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 3437-3443 -445, Online. Association for Compu- tational Linguistics.
Itiroo Sakai. 1961. Syntax in universal translation. In Proceedings of the International Conference on Ma- chine Translation and Applied Language Analysis, National Physical Laboratory, Teddington, UK.

---

Jingbo Shang, Liyuan Liu, Xiaotao Gu, Xiang Ren, Teng Ren, and Jiawei Han. 2018. Learning named entity lagger using domain-specific dictionary. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2054-2064, Brussels, Belgium. Association for Computational Linguistics.
Proceedings of the 58th Anmual Meeting the Asso- ciation for Computational Linguistics, pages 6470- 6476, Online. Association for Computational Lin- guistics.
Xinghua Zhang, Bowen Yu, Tingwen Liu, Zhenyu Zhang, Jiawei Sheng, Xue Mengge, and Hongbo Xu. 2021. Improving recognition with self-collaborative denoising Iearn- ing. In Findings of the Association for Computa- tional Linguistics: EMNLP 2021, pages 1518-1529, Punta Cana, Dominican Republic. Association for Computational Linguistics.
named entity
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. 2022. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Sys- tems, pages 1-19.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction the CoNLL-2003 shared task: I anguage-independent named entity recognition. In Proceedings of the Seventh Conference on Natu- ral Language Learning HLT-NAACL 2003, pages 142-147.
Kang Zhou, Yuepei Li, and Qi Li. 2022. Distantly su- pervised named entity recognition via confidence- based multi-class positive and unlabeled learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 7198-7211, Dublin, Ireland. Association for Computational Linguistics.
Jianing Wang, Chengyu Wang, Jun Huang, Ming Gao, and Aoying Zhou, 2023. Uncertainty-awar self- training for low-resource neural sequence labeling. arXiv preprint arxiv:2302.08659.
A Dataset
We considered and conducted the evaluation of our methods on the following datasets. Basic data statistics are also reported in Table 7.
Rui Wang, Xin Xin, Wei Chang, Kun Ming, Biao Li, and Xin Fan. 2019a. Chinese ner with height-limine limited constituent parsing. Proceedings of the AAAI Conference on Artificial Inielligece, 33(01):7100-117.
Youku It is sourced from the NLPCC-2020
Shared Task on AutolE and specifically involves caption text from YouKu vidcos (Yang et al., 2020). It focuses on three categories: TV, PER, and NUM. The data divided into three sets for training, development, and testing. Training dataset is an unlabelled corpus containing 10,000 samples. and the entities are labelled by string matching with the given entity lists. Entity lists with specific category, which may cover around 30% of entities appearing in the unlabelled corpus. Dev dataset contains 1,000 samples with full label, and 2,000 samples with full label in test dataset.
Zihan Wang, Jingbo Shang. Liyuan Liu, Lihao Lu, Ji- acheng Liu, and Jiawei Han. 2019b. CrighWeigh: Training named entity tagger from imperfect anno- tations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Process- ing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5154-5163, Hong Kong, China. Association for Computational Linguistics.
Songlin Yang and Kewei Tu. 2022. Bottom-up con- stituency parsing and nested named entity recogni- tion with pointer networks. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 2403-2416, Dublin, Ireland, Association for Com- putational Linguistics.
Weibo This dataset involves Chinese social me- dia messages (Peng and Dredze, 2015). An- notations were initially generated using Ama- zon Mechanical Turk and subsequently refined by amalgamating labels from different Turkers, which introduced some inconsistencies and errors. The dataset was manually corrected by Peng and Dredze (2017), resulting in cleaner version. It contains 1,890 messages sampled from Weibo be- tween November 2013 and December 2014. The sentences in training set, dev and test is 1,350, 270 and 270. The entity types include Geo- political, Location, Organization, and Person in both Name and Nominal forms. In the corrected version, partially annotated training set accounts
Xuefeng Yang, Benhong Wu, Zhanming Jie, and Yun- feng Liu. 2020. Overview of the NLPCC 2020 Shared Task: AutolE, pages 558-566.
Yaosheng Yang, Wenliang Chen, Zhenghua Li. Zhengqiu He. and Min Zhang. 2018 Distantly pervised NER with partial annotation learning and reinforcement learning. In Proceedings of the 27th International Conference guistics, pages 2159-2169, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Computational Lin-
Juntao Yu, Bernd Bohnet, and Massimo Poesio. 2020. Named entity recognition as dependency parsing. In

---

Data
Banch Size
LR
Trin Time
Inder Tims
Max Epochs
Fl aid
Youka
le-4
00:11:11
00:00005
20
41.50 =: 1.32
BEXT
Welbo
16
1c-4
0000:477
0200002
20
64,02 a 1.24
CONLLOS
1o-4
00:15:0
00:00.05
20
89.96 x 0.52
Partici Anmotation Settings
Youks
32
10.5
01:29:45
0200001
10
73.45 0.32
Weigitd PA (2019)
Welbo
32
1e.5
00:39:51
0000001
10
50.27 = 1.54
CONLLOO
06:11:35
02000g4
8L.9
Youki
67.87
NegSumpling (200))
00:19:1
60.00
CONILOOS
18
le.5
01:06007
000000
91.67
Youka
126
30.
01:3050
61.50 7.990
CIIE-MPU (2002)
Welho
16
5o-5
00:14:15
0000008
41.42 2.51
C.N
128
Se-5
00:15:40
03:8647
84.07 + 0.99
Incorrect Annotation Settings
Youka
32
10.5
00:5.07
0000001
50
45.20 0.38
BOND (2025)
Welbo
00:06:36
0200001
54.14
CONLLOS
01.09.17
NNCI (2011)
Weibo
02:18:34
020001
10
56 38
CONLLOS
le-5
05:57:24
0000005
10
79.07
= 1.84
Youks
le-4
08.26:52
0000001
50
S7.07
1004
(202)
Weits
le-d
00:25:20
03:0001
50
37.53
566
CONLLOS
100033
07:4437
(2022)
00:3:10
CONLLIOS
16
le-5
16:05:15
00:00.39
30
92.83 = 0.11
Enitity Setting
Youku
3e:5
01:16-11
020007
20.13
Partal-Trze (2021b)
Welbo
36:5
00:29:27
0000003
50, 43 2009
CUNILDOS
96
3e-5
01:56:09
00:00:08
30
92.5 0.23
Youku
3e-5
05:59-44
00:00441
75. 38
Ours
Welbo
3e-5
00:26:00
00000:34
60.64 2 0.781
CoNLLOS
96
36-5
08:11:39
00002:00
92.85 a: 0.15
Table 6: Hyperparameters and computational time for baselines and our model (corresponding to the main results in Table 1). Note that for the "Max Epochs", certain models involve both inner and outer loops, and in this analysis, we'v ve considered only the outer loops. Furthermore, the reported training time encompasses the cumulative time for external loops, along with the inference time for single iteration, expressed in hours, minutes and seconds.
for 27.9%, and incorrectly annotated training set makes up 8.49%.
Liitity c13)
Tox
cuity
sent
eatity
eatity
Youlu
5,299
2,000
Weibo
336.
1,370
1,350
301
270
414
270
CoNLLO3 This dataset comprises newswire from the Reuters RCVI corpus (Wang al., 2019b), tagged with four entity types (PER, LOC, ORG, MISC). It consists of standard training, de- velopment, and test sets. The statistics show sentence counts of 14,987, 3,466 and 3,684, though some records report 14,041, 3,250 and 3,453, as we did not exclude duplicates marked by" '-DOCSTARTT. Wang et al. (2019h) has iden- tified and corrected annotation mistakes in about 5.38% of test sentences.
CANLLOG
4 05.4
23.499
14,987
5,942
3,406
5,702
3,684
Taobao
29.397
6,000
4,941
998
4,806
1,000
Table 7: Data statisics. While Entity c(%) reveals the count of entity types along with their respective noisy ratios. Train, Validation and Test denote the quantities of entities sentences within each dataset.
the standard deviation in Table 6. On baselines, where we do finet-tuning, some hyperparameters were mannually tuned but most left at their default values. The final values for the mannually tuned and the time for train and infer are also provided in Table 6.
that
TaoBao This dataset relates the e-commerce domain and crawled and manually annotated by Jie et al. (2019). comprises sentences for training, development, and test sets with counts 6,000, 998 and 1,000 respectively, covering four distinct entity types: PATTERN, PRODUCT, BRAND, and MISC.
Models and Reproducibility
We provide average micro metrics over 5 seeds across each dataset in Table I. And we present


Reviews:
[{"review_id": "Ei4UTDPtwQ", "reviewer": "Reviewer_LrZJ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate your feedback and would like to provide a point-by-point response.\n\n**W1: Regarding error analysis of the results and ways to interpret what the method does beyond looking at F1 scores.**\n\n**Q1**: Thank you for your concern, which will be especially helpful in enhancing the manuscript. Indeed, in addition to the F1 scores, we studied additional metrics to explain the strengths of our method. Refer to Line 405, where we discover that our method is particularly good at recall, implying that it can mitigate the negative effect of mislabeled examples (“In addition, our method obtains a high recall rate, ranking first for Youku and second for Weibo, which demonstrates its robustness in mitigating false negatives in partial annotations”).\n\nThis claim was clarified in our revised version by using a more thorough comparison. Our method, in particular, raises the recall of a vanilla Tree-CRFs from 47.08 to 60.87 (+13.79%) on Weibo with partial annotations (Section 5.2), and from 58.77 to 61.70 (+2.93%) on Weibo with incorrect annotations (Section 5.3). The significant improvement in partial annotations shows that by incorporating uncertainty modeling within the CRFs, the model becomes more resistant to false negatives in annotations and resists its deleterious effects on training.\n\nWe've also added a new error analysis section to compare the results of our method and a standard Tree-CRFs. Interestingly, the entities found by our methods convers most of the entities found by vanilla Tree-CRFs (96.4%), and in 67.4% of the extra entities discovered by our method, we can identify an unannotated example (false negative) with a similar pattern in the training set (judge by human). This is another evidence that we can reduce the impact of false negatives in training.\n\n\n**W2: Regarding adding more details of the datasets.**\n\n**Q2**: Thank you for this insightful suggestion. In the appendix, we have added further explanations about the datasets, which can be summarized as follows:\n\n-\tYouKu Dataset: This dataset is sourced from the NLPCC-2020 Shared Task on AutoIE and specifically involves caption text from YouKu videos. It focuses on three categories: TV, person, and series. The data is divided into three sets for training, development, and testing. Train dataset: Unlabelled corpus containing 10000 samples, the entities are labelled by string matching with the given entity lists. Entity lists with specific category, which may cover around 30% of entities appearing in the unlabelled corpus. Dev dataset: 1000 samples with full label. Test dataset: 2000 samples with full label.\n\n-\tWeibo Dataset: This dataset involves Chinese social media messages and annotations. The annotations were created using Amazon Mechanical Turk and later refined by merging labels from different Turkers. This process led to some inconsistencies and errors. The dataset was manually corrected over a period of two years, resulting in a cleaner version. It contains 1,890 messages sampled from Weibo between November 2013 and December 2014. The sentences in training set, dev set and test set is 1350,270 and 270. The entity types include Geo-political, Location, Organization, and Person. In the corrected version, partially annotated training set accounts for 27.9%, and incorrectly annotated training set accounts for 8.49%.\n\n-\tCoNLL03 Dataset: This dataset comprises newswire from the Reuters RCV1 corpus, tagged with four entity types (PER, LOC, ORG, MISC). It consists of standard training, development, and test sets. The counts of sentences in these sets are 14,987, 3,466, and 3,684 respectively. There might be some statistics as 14041, 3250, 3453, because we didn't exclude the duplicates '-DOCSTART-'. Some papers identify and correct label mistakes in about 5.38% of test sentences.\n\n-\tTaoBao Dataset: This dataset is related to the e-commerce domain and is crawled and manually annotated. It includes sentences for training, development, and test sets with counts of 6,000, 998, and 1,000 respectively. The dataset encompasses four types: PATTERN, PRODUCT, BRAND, and MISC.\n\nThank you again.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Very elegant modeling approach that makes clever use of classical results and tools from ML and NLP.\n- Results demonstrate good performance on high-quality test sets as well as robustness to varying degrees of noise in the training set.\nThe authors add a thorough analysis on 4 different datasets as well as simulated data with different mislabeling conditions. The analysis also includes an ablation study for each component.\n- The manuscript reads well and is well organized.", "reasons_to_reject": "I thank the authors for addressing these concerns in their response:\n\n- While the performance numbers are convincing, there is no error analysis of the results, and no way to interpret what the method does beyond looking at F1 scores.  \n\n- The reporting in the experimental design section could be improved, the datasets used include details such as domain, size, estimation of noise in the dataset, distribution of NER labels, and most importantly, how was the high-quality version of each test set obtained and what is its size.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work proposes a new method for learning a NER tagger from very noisy training datasets, that either mislabel some entity mentions or fail to detect them. The authors demonstrate how to combine uncertainty estimation scores into a constituency parser that labels NER constituents. The method consists of two main components: a tree-based CRF for the parser, and a dropout-based uncertainty estimator. \nThese are trained using an iterative approach where each component's output serves to train the other in an iterative fashion.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lxZXUGos92", "reviewer": "Reviewer_TyYs", "summary": "", "questions": "see some suggestions made above (Reasons to Reject)", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful feedback and would like to respond in detail.\n\n**Q1. When simulating error annotations, do you operate on token or span level?**\n\n**A1**: Thank you for expressing concern. We take into account *span level* in both partial annotation simulations and incorrect annotation simulations (such as masking an entire entity for partial annotation evaluation and replacing an entity's label for incorrect annotation evaluation), which guarantees a pure simulation evaluation scenario.\nIn contrast, word-level masking/replacing may result in a mix situation (such as the Lincoln Memorial (LOC) being mistakenly annotated as Lincoln (PER)), making independent evaluation difficult. Moreover, given that such mixed cases are infrequent in real-world annotations (occurring in less than 2% of cases in CoNLL 2003 and Weibo datasets), we have opted for span-level simulation instead of word-level simulation.\n\n\n**Q2: Regarding the experimental setting and adding mean and standard deviation.**\n\n**A2**: Thank you for expressing concern. All of the results in our experiments are based on a 5-run average (as shown in the captions of Figures 5 and 6). We made it clearer in the new version by including this explanation in the \"Datasets and Evaluations\" section. In the upated we've also included standard deviation values of the F1 to the experiments, such as the 0.31 for the Weibo dataset (Table 1).\n\n\n**Q3: What is the goal of designing \"learning from pure noise\" experiments?**\n\n**A3**: Thank you for this question. The goal of designing \"learning from pure noise\" is to evaluate our model's ability to learn from *completely incorrect annotations (i.e., \"pure noise\")*. Assume we have a clean dataset and a second set of examples with completely incorrect entity labels. Then, it's common to believe that adding such incorrect examples to the clean dataset will impede learning and degrade performance. However, according to the experimental results, our model performs better when \"pure noise\" is added. This suggests that our model can learn from even pure noise, and that samples with completely incorrect labels can provide some supervision for learning.\n\nLastly, we highly value the suggestions for typos, grammar, style, and presentation improvements and have incorporated them into the updated version. Thanks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Annotation noise is a very important problem for NER and the paper presents an interesting approach to training noise-robust NER models. It is overall well written, structured and easy to follow the argumentation. The idea of alternating between uncertainty evaluation and model updates is inspiring and well described.\nI appreciate especially that they used real-world datasets that happen to exist with noisy and clean variants/splits instead of just simulating noise artificially! But of course, also the synthetic experiments are very interesting and striking.\nThe experimental results are well presented and discussed, I appreciate the large number of competitive models that the authors compare their methods to.", "reasons_to_reject": "I do not have major concerns about the paper. One thing that could be made clearer is that, when e.g. categorizing and simulation error types, they operate on token or span level (i.e. when adding synthetic partial errors: do whole mentions get assigned the O label or only random tokens?). \nIt seems that in the results tables (in contrast to those presented by the figures), they only present one specific model run. Given that those numbers do not show that much of deviations, it might be better to add mean and standard deviation of several runs.\nThe ablation experiments could benefit from a bit more care in their description, especially the one about \"Learning from Pure Noise\" which in my opinion was a bit hard to understand. E.g. make more clear what makes the \"Pure Noise\" experiments different to the experiments before. Also Sec. 6.2 Case Study and its figure should be described more clearly.", "questions_to_authors": "see some suggestions made above (Reasons to Reject)", "paper_topic_and_main_contributions": "The paper presents an approach for training a noise-robust NER model that can learn well even in scenarios with extremely high noise rates in the training data. It consists of mainly two parts: Learning to assess uncertainty of the ground truth labels (via variance of predictions of different models when using MC-Dropout) and then training a tree-structured Conditional Random Fields (CRFs) that includes these uncertainty nodes. They apply an iterative co-learning mechanism with alternating between running the Uncertainty Evaluation and training the Tree CRFs parsing model, which helps the model to more and more train well on the real data and get better at detecting noise.\n\nThe authors present experiments on three real-word NER datasets where a noise-cleaned test set is available (Youku, Weibo, CoNLL03). Their method outperforms the BERT baseline as well as several other noise-robust models.\n\nThey also present experimental results on synthetic variants of these dataset where they differentiate between partial annotation errors (some entities are missing a label) and incorrect annotation errors (wrong label) and simulate extreme noise scenarios, showing that the model is especially robust when trained on partial annotations (meaning some labels are simply removed) compared to the other models. Interestingly (and maybe not discussed enough) when trained on high rates of incorrect annotation (labels are switched) the model shows surprisingly weaker quality (but still beating the rest of models).\n\nSome ablation experiments are presented as well, showing the necessity of uncertainty evaluation and their co-learning mechanism.\n", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "* page 1, line 032: \"Loc Angeles Lackers\" > \"Los Angeles Lakers\"\n* page 2, line 149: \"we structures\" > \"we structure\"\n* page 4, line 263: check grammar\n* Figure 8: \"Glod\" > \"Gold\"\n"}, "extra_text": {}}, {"review_id": "WZMvyfgM94", "reviewer": "Reviewer_hCvH", "summary": "", "questions": "Question A: In section 3.2, the proposed method (MC-Dropout) seems to measure the uncertainty of model predictions rather than the reliability of the annotated label (Figure 2)? would you please clarify how these two concepts relate to each other?\n\nQuestion B: Line 337: the way of creating incorrect annotation considers only entity type prediction error; suggest considering entity boundary errors as well, or explaining why boundary errors are not considered.\n\nQuestion C: For MC-Dropout, different epochs are searched for MC-Dropout (line 337). Would you please discuss the impact of the number of epochs and clarify whether a similar grid search has been applied to other methods in Table 4?\n\nQuestion D: Can you explain why Conf-MPU's performance on Taobao (Figure 5, 6) is much worse than other methods even when the ratio is 0, whereas, on other datasets, the gap between Conf-MPU and other methods seems to be smaller?\n\nQuestion E: Line 506: not sure what 'pure noises' mean, especially when 'noise ratio' is also introduced in Figure 7?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We value your insightful feedback and would like to provide a detailed response addressing each point.\n\n*For the weaknesses*\n\n**W1: A clarity about where the main improvements come from.**\n\n**A1**: Thank you for raising this concern. Our method can be thought of as an organically ensembled Tree-CRF model with an uncertainty evaluation module. To see where the main improvements come from, we refer to experiments in Tables 2 and 3, where, when compared to a vanilla Tree-CRFs, our method improves recall from 47.08 to 60.87 (+13.79%) on Weibo with partial annotations (Section 5.2), and from 58.77 to 61.70 (+2.93%) on Weibo with incorrect annotations (Section 5.3). The large improvement in partial annotations implies that by including uncertainty modeling within the CRFs, the model becomes more resistant to false negatives and resists their negative effects on training.\n\nTable 4 and 5 evaluate several uncertainty evaluation and co-training mechanism, but we believe that the new way to incorporate uncertainty assessment into a Tree-CRE parsing method is the most essential component for improvement. Results in Table 4 also support this claim. For example, even when augmenting vanilla CRFs with Softmax Probability (which is not a principled uncertainty measurement), the outcomes can be improved.\n\nWe hope the above explanation clarifies the concern. Thank you.\n\n\n\n**W2: Regarding experimental settings.**\n\n**A2**: Thank you for expressing this concern. All of the results in our experiments are based on a 5-run average (as shown in the captions of Figures 5 and 6), and we tuned baselines using a grid search then perform the same 5-run average evaluation. As we discovered in our research, some baselines are unsuitable for real-world NER datasets. Conf-MPU, for example, uses external knowledge bases to generate noisy labels, but it is difficult to obtain related knowledge bases for real-world datasets like TaoBao, where the majority of entities are emergent names for items.\n\nIn the revised version, we have provided the aforementioned explanation in the \"Datasets and Evaluations\" section, and we have also included standard deviation values of the F1 to the experiments, such as the 0.31 for the Weibo dataset (Table 1). \n\n\n*For the questions*\n\n**Q1: MC-Dropout seems to measure the uncertainty of model predictions rather than the reliability of the annotated label? would you please clarify how these two concepts relate to each other?**\n\n**A1**: We really appreciate this insightful question. We believe that the two concepts can be distinguished by using different modeling frameworks to explain them.\n-\tFor example, if we want to model \"reliability of the annotated label\", the best way is to use generative modeling and associate a latent variable to represent the ground-truth label. Then we should infer the label from the observed data, and the discrepancy between the inferred and observed labels represents the annotated label's reliability. \n \n-\tIn contrast, the goal of discriminative modeling methods (like neural network-based methods) is to directly predict the labels without requiring a ground-truth inferring procedure. In this way, we have to measure the uncertainty of the model itself, such as using the MC-Dropout method.\n\nThanks.\n\n\n**Q2: Explaining why boundary errors are not considered.**\n\n**A2**: Thank you for this concern. In incorrect annotation simulations, we use span-level replacement and do not explore scenarios with boundary errors. One explanation is that entities with boundary errors are relatively rare in real-world annotations (occurring in fewer than 2% of cases in CoNLL 2003 and Weibo datasets), therefore an over-strength on such cases may make the evaluation unrealistic. The second reason is that we wish to separate partial annotation simulations from incorrect annotation simulations, however the boundary error may be a combination of two annotation issues, making a separated evaluation difficult.  \n\n\n**Q3: Regarding giving the impact of the number of epochs in MC-Dropout and clarifying whether a similar grid search has been applied to other methods in Table 4?**\n\n**A3**: Thank you for this question. Do you mean the number of forward predictions (i.e., T in L268) when you refer \"number of epochs in MC-Dropout\"? Indeed, T is an MC-Dropout hyper-parameter; a large T makes the results more accurate but takes longer. In our approach, we use the smallest T value that is sufficient for uncertainty evaluation to achieve a speed-effectiveness balance (in fact, it is good to set T as 10 for Youku as well, and the results decrease about 0.7% in F1). However, none of the other methods in Table 4 have an equivalent hyper-parameter, thus we do not execute a grid search on them. Thanks.\n\n\n**Q4: Can you explain why Conf-MPU's performance on Taobao (Figure 5, 6) is much worse than other methods even when the ratio is 0, whereas, on other datasets, the gap between Conf-MPU and other methods seems to be smaller?**\n\n**A4**: Thank you for this question. One reason of the degraded performance of Conf-MPU is that it uses external knowledge bases to generate noisy labels, but it is difficult to obtain related knowledge bases for real-world datasets like TaoBao, where the majority of entities are emergent names for items\n\n**Q5: Line 506: not sure what 'pure noises' mean, especially when 'noise ratio' is also introduced in Figure 7?**\n\n**A5**: Thank you for this question. In \"learning from pure noise\", we aim to evaluate our model's ability to learn from *completely incorrect annotations (i.e., \"pure noise\")*. Assume we have a clean dataset and a second set of examples with completely incorrect entity labels. Then, it's common to believe that adding such incorrect examples to the clean dataset will impede learning and degrade performance. However, according to the experimental results, our model performs better when \"pure noise\" is added. This suggests that our model can learn from even pure noise, and that samples with completely incorrect labels can provide some supervision for learning.\n\n**Regarding typos, grammar style, and presentation improvements.**\n\nWe highly appreciate the suggestions for revisions of typos, language style, and presentations and have incorporated them into the revised edition. In particular, in Figure 1, we have added a footnote explaining the incorrect annotation and its connection to the nested NER. We fixed typos and added a time comparison to compare the effectiveness of different methods. We have also included the missing reference for completeness.\n\nThanks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A method of training NER model from imperfect annotations; empirical results show that the proposed method can effectively address different types of annotation errors.", "reasons_to_reject": "There is a lack of clarity about where the main improvements come from (Table 4 and 5 show that the MC-Dropout and Co-learning are slightly better than alternatives); ~I also have a concern that the authors seem to report the best run results of their propose method but the baselines are not well-tuned~ (delete after rebuttal)", "questions_to_authors": "Question A: In section 3.2, the proposed method (MC-Dropout) seems to measure the uncertainty of model predictions rather than the reliability of the annotated label (Figure 2)? would you please clarify how these two concepts relate to each other?\n\nQuestion B: Line 337: the way of creating incorrect annotation considers only entity type prediction error; suggest considering entity boundary errors as well, or explaining why boundary errors are not considered.\n\nQuestion C: For MC-Dropout, different epochs are searched for MC-Dropout (line 337). Would you please discuss the impact of the number of epochs and clarify whether a similar grid search has been applied to other methods in Table 4?\n\nQuestion D: Can you explain why Conf-MPU's performance on Taobao (Figure 5, 6) is much worse than other methods even when the ratio is 0, whereas, on other datasets, the gap between Conf-MPU and other methods seems to be smaller?\n\nQuestion E: Line 506: not sure what 'pure noises' mean, especially when 'noise ratio' is also introduced in Figure 7?", "paper_topic_and_main_contributions": "This paper describes a method that focuses on training NER model on noisy annotations.\nThe authors frame the NER as a constituency parsing problem where each node in the constituency tree has a label representing the entity type of the textual span. To address annotation noise, the authors introduce uncertain nodes in the constituency tree. They use tree-structured CRFs to average uncertainty and MC-Dropout for evaluating uncertainty. Four NER datasets were used to evaluate the effectiveness of the proposed method. The experimental results show that the proposed method outperforms several baselines that focus on different types of noisy labels.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1]\t Lukas Lange et al., \"Feature-dependent confusion matrices for low-resource ner labeling with noisy labels\", in EMNLP-IJCNLP, 2019."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Section 1 and Figure 1: suggest giving a brief definition of these two terms: `partial annotation` and `incorrect annotation`; because it seems that the example involving `Newcobe` is a missing (false negative) annotation; and the example involving `Los Angeles Lakers` can be attributed to the nested NER [1]; Arguably, `Los Angeles` is a partial annotation rather than an incorrect annotation.\n\nLine 67: suggest adding the complete name (Monte Carlo?) when the abbreviation `MC' is first mentioned\n\nLine 371: 1E-5 -> 1e-5, same changes applied to line 373\n\nSuggest providing a summary table of used datasets, highlighting the different characteristics of noises (e.g., the source, noisy level). This may help the readers understand why very different patterns are observed on CoNLL03 and others.\n\nLine 505: an interesting\n\nSuggest adding efficiency analysis to understand the training and inference cost of the proposed method and alternatives.\n\n[1]\t Nicky Ringland et al., \"NNE: A Dataset for Nested Named Entity Recognition in English Newswire\", in ACL, 2019."}}]

Author rebuttal:

Meta-review: The reviews have a consensus on positive evaluations for both soundness and excitement.

Regarding excitement, the proposed method of training an NER model from noisy data is novel.  In the line of framing NER as constituency parsing, the method introduces uncertain nodes in the constituency tree and trains tree-structured CRFs with MC-Dropout to measure uncertainty.

As for soundness, the paper provides extensive comparisons with several baselines on 4 datasets and their noisy variants with respect to 3 annotation error types.  More descriptions on the quality of test sets (e.g., the corrections that were made) would improve the soundness of evaluation.
Decision: Accept-Main