Empirical Analysis of Unlabeled Entity Problem in Named Entity Recognition
Authors: Yangming Li
lemao liu
Shuming Shi
Venue: ICLR 2021 Poster (2021)
Keywords: Named Entity Recognition
Unlabeled Entity Problem
Negative Sampling

Abstract: In many scenarios, named entity recognition (NER) models severely suffer from unlabeled entity problem, where the entities of a sentence may not be fully annotated. Through empirical studies performed on synthetic datasets, we find two causes of performance degradation. One is the reduction of annotated entities and the other is treating unlabeled entities as negative instances. The first cause has less impact than the second one and can be mitigated by adopting pretraining language models. The second cause seriously misguides a model in training and greatly affects its performances. Based on the above observations, we propose a general approach, which can almost eliminate the misguidance brought by unlabeled entities. The key idea is to use negative sampling that, to a large extent, avoids training NER models with unlabeled entities. Experiments on synthetic datasets and real-world datasets show that our model is robust to unlabeled entity problem and surpasses prior baselines. On well-annotated datasets, our model is competitive with the state-of-the-art method.

Published as a conference paper at ICLR 2021
EMPIRICAL ANALYSIS OF UNLABELED ENTITY PROB- LEM IN NAMED ENTITY RECOGNITION
Yangming Li, Lemao Liu, & Shuming Shi
Tencent Al Lab (newmanli i, redmond1 fu, shumingshi
ABSTRACT
In many scenarios, named entity recognition (NER) models severely suffer from unlabeled entity problem, where the entities of sentence may not be fully an- notated. Through empirical studies performed on synthetic datasets, we find two causes of performance degradation. One is the reduction of annotated entities and the other is treating unlabeled entities as negative instances. The first cause has less impact than the second one and can be mitigated by adopting pretraining language models. The second cause seriously misguides model in training and greatly affects its performances. Based on the above observations, we propose a general approach, which can almost eliminate the misguidance brought by unla- beled entities. The key idea is to use negative sampling that, to a large extent, avoids training NER models with unlabeled entities. Experiments datasets and real-world datasets show that our model is robust to unlabeled entity problem and surpasses prior baselines. On well-annotated datasets, our model is competitive with the state-of-the-art method"
synthetic
INTRODUCTION
Named entity recognition (NER) is an important task in information extraction. Previous methods typically cast into a sequence labeling problem by adopting 1OB tagging scheme (Mesnil et al.) 2015 Huang et al. 2015 Ma & Hovy
2016 Akbik et al. 2018 Qin et al.) 2019). A representative model is Bi-LSTM CRF (Lample et al.
2016). The great success achieved by these methods benefits from massive correctly labeled data. However, in some real scenarios, not all the entities in the
training corpus are annotated. For example, in some NER tasks (Ling & Weld 2012). the datasets contain too many entity types mention may be associated with multiple labels. Since manual annotation on this condition is too hard, some entities are inevitably neglected by human annotators. Situations in distantly supervised NER (Ren et al. 2015 Fries al. 2017) are even more serious. To reduce handcraft annotation, distant supervision (Mintz et al.) 2009) is applied to automatically produce labeled data. As result, large amounts entities in the corpus are missed due to the limited coverage of knowledge resources. We refer this to unlabeled entity problem, which largely degrades performances of NER models.
There are several approaches used in prior works to alleviate this problem. Fuzzy CRF and Au- toNER (Shang et al.) 20186) allow models learn from the phrases that may be potential en- tities. However, since these phrases are obtained through distantly supervised phrase mining method (Shang et al., al, 2018a), many unlabeled entities the training data may still not be recalled. In the context of only resorting to unlabeled corpora and entity ontology, Mayhew et al. (2019): Peng ct al.] (2019) employ positive-umlabeled (PU) learning (Li & Liu 2005) to unbiasedly and consis- tently estimate the task loss. In implementations, they build distinct binary classifiers for different labels. Nevertheless, the unlabeled entities still impact the classifiers of the corresponding entity types and, importantly, the model can't disambiguate neighboring entities. Partial CRF (Tsuboi et al. 2008) is an extension of commonly used CRF (Lafferty et al.] 2001) that supports learning from incomplete annotations. Yang al. (2018); Nooralahzadeh al. (2019); Jie al. (2019) use it to circumvent training with false negatives. However, as fully annotated corpora are still required
Our source code is available http://gghhaaccomm/c//aass//eset

---

Published as a conference paper at ICLR 2021
to get ground truth training negatives, this approach is not applicable the situations where little or even no high-quality data is available.
In this work, goal is study what are the impacts of unlabeled entity problem the models and how to effectively eliminate them. Initially, we construct some synthetic datasets and introduce degradation rates. The datasets are constructed by randomly removing the annotated named entities in well-annotated datasets, e.g.. CoNLL-2003 (Sang & De Meulder (003), with different probabil- ities. The degradation rates measure how severe an impact of unlabeled entity problem degrades the performances of models. Extensive studies are investigated on synthetic datasets. We find two causes: the reduction of annotated entities and treating unlabeled entities as negative instances. The first cause is obvious but has far fewer influences than the second one. Besides, it can be mitigated well by using a pretraining language model, like BERT (Devlin et al. 2019)), as the sentence en- coder. The second cause seriously misleads the models in training and exerts great negative impact on their performances. Even less severe cases, it can sharply reduce the F1 score by about 20%. Based on the above observations, we propose novel method that is capable of eliminating the mis- guidance unlabeled entities in training. The core idea is to apply negative sampling that avoids training NER models with unlabeled entities.
Extensive experiments have been conducted to verify the effectiveness of our approach. Studies on synthetic datasets and real-world datasets (c.g.. EC) show that our model well handles unlabeled entities and notably surpasses prior baselines. On well-annotated datasets (e.g., CoNI.L-2003),, our model is competitive with the state-of-the-ar method.
2
PRELIMINARIES
In this section, we formally define the unlabeled entity problem and briefly describe a strong base- line, BERT Tagging (Devlin et al. 2019), used in empirical studies.
2.1 UNLABELED ENTITY PROBLEM
[x11, U2. [n]] and the annotated named entity set as (y1, 3/2. ... )m )- n the sentence length and m is the amount of entities. Each member y/ of set y is tuple (kk.kk 4x). (ik.je) is the span of an entity which corresponds to the phrase Xia.de = [T4g 1i Tt.. and 45 is its label. The unlabeled entity problem is defined as, due to the limited coverage of machine annotator the negligence of human annotator, some ground truth entities ý of the sentence x are not covered by annotated entity set y.
We denote an input sentence
For instance, given a sentence [Jack, and, Mary, are, from, New, York] and labeled entity set = ((1, 1, }.R)]. unlabeled entity problem is that some entities, like (6. 7, LOC), are neglected by annotators. These unlabeled entities denoted y = {(3, 3, PER), (6, 7, LOC)).
2.2 BERT TAGGING
BERT Tagging is present in Devlin et al. (2019), which adopts IOB tagging scheme, where each token z; in a sentence x is labeled with a fine-grained tag, such as B-ORG, I-LOC, or O. Formally, its output is a n-length label sequence z = [a1, "2, =.
Formally, BERT tagging firstly uses BERT to get the representation h; for every token o;;
hy h2 h BERT(x).
Then, the label distribution qi is computed as Softmax (Wh)) In training, the loss is induced as Σ Eisien log qu [2]]. At test time, it obtains the label for each token 2; by arg max q.
3 EMPIRICAL STUDIES
To understand the impacts of unlabeled entity problem, we conduct empirical studies over multiple synthetic datasets, different methods, and various metrics.
2

---

Published as a conference paper at ICLR 2021
Topsing lagging
Tagging
Mingoddaas BRMC:
Erosica Robc
O.C.
044
BEAT Tooging
20- BEAT Tweging Adjucted
02-
LSTN Topging
LSTN" "aceing Adj.stoc Loss
0.0
11
02
0.1
ns De O7 0.0
03
07
033
05
08
03
00
5.0
0.
05 04
05
07
Masking Probabilitt
Masking Probubility
Misking Probubility
Figure 1: The empirical studies conducted on CoNLL-20003 dataset.
10
BERT Tagging
SERT tagging
LSTN "acging
LSTH Thgging
0.8
Miggrshare Robl
Famsion
504
8EK Togghng Tepping
Tagoing
Togaing Adjusted
an
03
Masking Probability
Masking Probubility
Mooking Probabiliny
Figure The empirical studies investigated on OntoNotes 5.0 dataset.
3.1 PREPARATIONS
Synthetic Datasets. We use synthetic datasets to simulate poorly-annotaie datasets that contain unlabeled entities. They obtained by randomly removing the labeled entities of well-annotated datasets with different masking probabilities p. The material datasets are CoNI.L-2003 (Sang & De Meulder (003) and OntoNotes 5.0 (Pradhan al. 2013). The probabilities p are respectively set as 0.0, 0.1, 0.2...... 0.9.. In this way, 2 x 10 synthetic datasets are constructed.
Methods. We adopt two models. One of them is BERT Tagging, which has long been regarded a strong baseline. The other is LSTM Tagging that replaces the original encoder (i.e., BERT) of BERT Tagging with LSTM (Hochreiter & Schmidhuber (197). We use to study the effect of using pretraining language model. To explore the negative impact brought by unlabeled entities in training, we present an adjusted training loss for above two models:
Σ 1S
log (1l=))
Σ E U.J.
log qu (=t)))
The idea here is to remove the incorrect loss incurred by unlabeled entities. Note that missed entity set is reachable in synthetic datasets but unknown in real-world datasets.
Metrics. Following prior works, the Fl scores of models are tested by using conlleval scrip We also design two degradation rates to measure the different impacts of unlabeled entity problem. One erosion rate ap and the other is misguidance rate Sp:
fo
fp
ap
B
88
For synthetic dataset with the masking probability being p. fp and fü are the Fl scores of a model and its adjusted version, respectively. Note that / corresponds to p = 0. Erosion rate a, measures how severely the reduction of annotated entities degrades the Fl scores of model. Misguidance rate Bp measures how scriously unlabeled entities misguide the model training.

---

Published as a conference paper at ICLR 2021
h
X/2
Sequence Encoder (BERT or LSTM)
hz
$1,3
MLP
01
ft
X4
h4
Figure 3: This demonstrates how our model scores possible entities.
3.2 OVERALL ANALYSIS
The left parts of Fig. and Fig. 2 show the results of empirical studies, where we evaluate the Fl scores of BERT Tagging and LSTM Tagging 20 synthetic datasets. From them, we can draw the following observations. Firstly, the significant downward trends of solid lines confirm the fact that NER models severely suffer from unlabeled entity problem. For example, by setting the masking probability as 0.4, the performance of LSTM Tagging decreases by 33.01% on CoNLL- 2003 and 19.58% on OntoNotes 5.0. Secondly, in contrast, the dashed lines change very slowly, indicating that the models with adjusted training loss (see Eq. (2) are much less influenced by the issue. For instance, when masking probability 0.7, adopting adjusted loss preserves the scores of BERT Tagging by 41.04% on CoNLL-2003 and 57 57.38% OntoNotes 5.0. Lastly, for high masking probabilities, even though the negative impact unlabeled entities eliminated by adjusting the training loss, the performance still declines to certain extent. For example, when masking probability is set as 0.8, the Fl scores of adjusted LSTT Tagging decrease by 31 36.6% on CONLL-20003 and 17.22% OntoNotes 5.0.
Reduction of Annotated Entities. From the last observation, can infer that cause of perfor- mance degradation the reduction of annotated entities. In the middle parts of Fig. [ and Fig. [ we plot the change of erosion rates Op (see Eq. (2) with respect masking probabilities. We can that its impact is not very serious when in low masking probabilities but can't It be neglected when in high ones. Besides, using pre- pre-training language models greatly mitigates the issue. As an example, when the probability is 0.8, on both CoNLL-2003 and OntoNotes 5.0, the erosion rates of adjusted BERT Tagging are only about half of those of adjusted LSTM Tagging.
Misguidance of Unlabeled Entities. From the last two observations, we can conclude that the primary cause is treating unlabeled entities negative instances, which severely misleads the mod- els during training. To better understand it, in the right parts of Fig. and Fig. 2
we plot the change of misguidance rates 3p (see Eq. (3) with masking probabilities. These rates are essentially the percentage decreases of Fl scores. From them, we can see that the impact of misguidance very much serious even when in low masking probabilities.
METHODOLOGY
Motivated by Sec. 3.2, we present model that is robust to unlabeled entities.
4.1 SCORING MODEL WITH BERT
Based on the findings in Sec, 3.2 we use BERT as the default encoder to mitigate the reduction of annotated entities. Specifically, given sentence we firstly obtain the token representations h; with Eq. T Then, we get the representation for every phrase X,,j as
Sij = h + hy ① (h. hy)
② (, @ hy),
(4)
where ① is column-wise vector concatenation and O is element wise vector product. The design here is mainly inspired by Chen al. (2017).
4

---

Published as a conference paper at ICLR 2021
Finally, multi-layer perceptron (MLP) computes the label distribution oz, for span (i,j):
i Softmax( U tanh
(5)
The term oigi the predicted score for entity (1,
4.2 TRAINING NEGATIVE SAMPLING
we know that regarding all the unlabeled spans as negative instances certainly de- grades the performances of models, since some of them may be missed entities. Our solution this issue is negative sampling. Specifically, we randomly sample a small subset of unlabeled spans as the negative instances induce the training loss.
From Sec. 3.2
Given the annotated entity set y, we firstly get all the negative instance candidates as
{i.,j.) (i.j.)) $ y, sis n, E C},
(6)
where L is the label space and O is the label for non-entity spans.
Then, we uniformly sample subset ý from the whole candidate set. The size of sample set ý is [A n1.0 < 1, where [7 is the ceiling funetion.
Ultimately, a span-level cross entropy loss used for training is incurred as
- -log(o:,j1)) +
Σ - log log(orr.
(.j-eyy
Negative sampling incorporates some randomness into the training loss, which reduces the risk of training a NER model with unlabeled entities.
4.3 INFERENCE
At test time, firstly, the label for every span (i,j) )) is obtained by arg max, oz,] [4. Then, we select the ones whose label is not as predicted entities. When the spans of inferred entities intersects, preserve the one with the highest predicted score and discard the others.
The time complexity inference O(n²), which is majorly contributed by the span selection pro- cedure. While this seems bit higher compared with our counterparts, find that, in practical use, its running time is far less than that of the forward computation of neural networks. The algorithm for inference is greedy yet effective. In experiments, find that the probability of our heuristic selecting wrong labeled span when resolving the span conflict is very low.
DISCUSSION
We show that, through negative sampling, the probability of not treating specific missed entity in n-length sentence as the negative instance is larger than 1
n
II
(1-
[mm]
m
n(n+1)
2
m
[nn]
(8)
1
1
2
(n++1)
n(n+1)
n
Here n(x+1) m is the amount of negative candidates. Besides, we use the facts, À 1, m S n, 2 and (1 - 2) 2 - n2, 0 < 2 < 1, during the derivation.
Note that the above bound is only applicable the special case where there just one unlabeled entity in sentence. We remain the strict proof for general cases to future work.
6 EXPERIMENTS
We have conducted extensive experiments on multiple datasets to verify the effectiveness of our method. Studies synthetic datasets show that our model can almost eliminate the misguidance
5

---

Published as a conference paper at ICLR 2021
CoNLL-20003
OntoNotes 5.0
Masking Prob.
BERT Tagging
Our Model
BERT Tagging
Our Model
0.1
90.71
91.37
87.69
89.20
0.2
89.57
91.25
86.86
89.15
0.3
88.95
90.53
84.75
88.73
0,4
82.94
89.73
82.55
88 20
0.5
78.99
89.22
71.07
88.17
0.6
63.84
87.65
58.17
87.53
Table 1: The experiment results on two synthetic datasets.
0.35
BERT Tagging
BERT Tagging
0.30
Our Model
Our Model
Misgnidanes Rate
0.25
0.1
0.2
03
344
0.5
0.6
0.1
0.2
0.3
as
06
Masking Probability (CoNIIL-2003)
Masking Probability (OntoNotes 5.0)
Figure 4: The misguidance rates of BERT Tagging and our model.
brought by unlabeled entities in training. On real-world datasets, our model has notably outper- formed prior baselines and achieved the state-of-the-ars performances. On well-annotated datasets, model is competitive current state-of-Ne ofethe-rrt method.
6.1 SETTINGS
The setup of synthetic datasets and well-annotated datasets are the same as what we describe in Sec. 3.1 For real-world datasets, use EC and NEWS, both of which are collected by Yang et al. (2018). EC e-commerce domain, which has 5 entity types: Brand, Product, Model, Material, and Specification. The data contains 2400 sentences tagged by human annotators and are divided into three parts: 1200 for training, 400 for dev, and 800 for testing. Yang et al. (2018) also construct entity dictionary of size 927 and apply distant supervision 2500 sentences for training. NEWS from MSRA dataset [Levow] 2006). Yang et al. (2018) only adopt the PERSON entity type. Training data of size 3000, dev data of size 3328, and testing data of size 3186 are all sampled from MSRA. They collect an entity dictionary of size 71664 and perform distant supervision on the rest data obtain extra 3722 training cases by using the dictionary. Both EC and NEWS contain a large amount of incompletely annotated sentences, and hence naturally suffer from the unlabeled entity problem.
corpus obtain additional
We adopt the same dyper-paramett configurations of neural networks for all the datasets. L2 reg- ularization and dropout ratio are respectively set as 10-5 and 0.4 for reducing overfit. The dimension of scoring layers is 256. Ratio À is set as 0.35. When the sentence encoder is LSTM, we set the hidden dimension as 512 and use pretrained word embeddings (Pennington et al. 2014 Song et al.) 2018) initialize word representations. We utilize Adam (Kingma & Ba 2014 as the optimization algorithm and adopt the suggested hyper pyper-parameters. At evaluation time, we convert the predictions of our models into IOB format and use conlleval script to compute the Fl score. In all the experiments, the improvements our models over the baselines are statistically significant with rejection probabilities smaller than 0.05.

---

Published as a conference paper at ICLR 2021
Method
CoNLL-2003
OntoNNoes 5.0
Flair Embedding (Akbik et al. 2018
93.09
89.3
BERT-MRC
et al. 2020a
93.04
91.11
HCR w/ BERT (Luo al. 2020)
93.37
90.30
BERT-Biaffine Model Yu et al. 2020)
93.5
91.3
Our Model
93.42
90.59
Table 2: The experiment results on well-annotated datasets.
Method
EC
NEWS
String Matching via Ontology
44.02
47.75
BiLSTM CRF
54.59
69.09
Yang et al. 2018
BiLSTM + CRF w/ RL
56.23
73.19
BiLSTM + Partial CRF
60.08
78.38
BiLSTM + Partial CRF w/ RL.
61.45
79.22
Jie al. 2019
Weighted Partial CRF
61.75
78.64
Nooralahzadeh al. (0019)
BiLSTM Partial CRF w/ RL
63.56
80.04
Our Model
66.17
85.39
This Work
Our Model w/o BERT, w/ BiLSTM
64.68
82.11
Table 3: The experiment results real-world datasets.
6.2 RESULTS ON SYNTHETIC DATASETS
In this section, our model is compared with BERT Tagging on the synthetic datasets of the masking probabilities being 0.1, 0.2,... 0.6. From Table we can get two conclusions. Firstly, our model significantly outperforms BERT Tagging, especially in high masking probabilities. For example, on CoNLL-2003, our Fl scores outnumber those of BERT Tagging by 1.88% when the probabílity is 0.2 and 27. 16.16% when the probability is 0.6. Secondly, our model is very robust to the unlabeled entity problem. When increasing the masking probability from 0.1 to 0.5, the results of our model only decrease by 2.35% on CoNLL-2003 and 1.91% on OntoNotes 5.0.
Fig. demmonstrates the misguidance rate comparisons between BERT Tagging and our models. The way to adjust our model is reformulating Eq. 7 by defining the negative term via ji O) AI (i, & yyý} rather than the negatively sampled y. The idea here is to avoid the unlabeled entities being sampled. From Fig. 4 we can discover that, in all masking probabilities, the misguidance rates of our model are far smaller than those BERT Tagging and are consistently lower than 2.50%. These indicate that, in training, model indeed eliminates the misguidance brought by unlabeled entities to some extent.
6.3 RESULTS ON FULLY ANNOTATED DATASETS
We additionally apply our model with negative sampling on the well-annotated datasets where the issue of incomplete entity annotation not serious. As shown Table the scores of our model are very close to current best results. Our model slightly underperforms BERT-Biaffine Model by only 0.09% on CoNLL-2003 and 0.78% on OntoNotes 5.0. Besides, our model surpasses many other strong baselines. On OntoNotes 5.0, our model outperforms HCR w/ BERT by 0.32% and Flair Embedding by 1.44%. On CoNEL-2003, the improvements of F1 scores are 0.41% over BERT-MRC and 0.35% over Flair Embedding. All these results indicate that model is still very effective when applied to high-quality data.
6.4 RESULTS REAL-WORLD DATASETS
For two real-world datasets, large portion of training data is obtained via distant supervision. As stated in Yang et al. 2018), the scores of string matching through an entity dictionary are notably declined in terms of the low recall scores, although its precision scores are higher than those of other methods. Therefore, wnlabeled entity problem serious the datasets. As shown in Table 3
the baselines come from three works (Yang al. 2018 Nooralahzadeh al. 2019 Jie et al.

---

Published as a conference paper at ICLR 2021
90
89
88
88
67
87
86
86
85
Masking Frob.: D.4
Masking Prot: 0
Masking
Macking
85
84
Masking Frob.:
Masking Prol:.:
Masking Prob. 0.7
Masking Prot:.:
Rd
0o
W5
0.4
Rale Negative Sampling (CoNLL-20033)
Rate in Negative Sampling (OntoNotes 5.0)
Figure 5: The results of our models with different ratio À synthetic datasets.
CoNLL-2003
OntoNotts 5.0
Metric
BERT Tagging
LSTM Tagging
BERT Tagging LSTM Tagging
Erosion Rate Opp
-0.94
0.90
-0.85
-0.82
Misguidance Rate Bp
-1.0
0.96
-1.00
-0.98
Table 4: The PCCs between Fl score and degradation rates.
2019). Yang et (2018) use Partial CRF circumvent all possible unlabeled entities and utilize reinforcement learning (RL) adaptively skip noisy annotation. Tie et al. (2019) andoralllahzad al.) 2019) respectively improve Partial CRF and the policy of RL. All the F1 scores of baselines copied from Yang al. (2018); Nooralahzadeh et al. (2019), except for that of Weighted Partial CRE, which obtained by rerunning its open-source code
Our model has significantly outperformed prior baselines and obtained new state-of-the-an results. Compared with prior best model (Nooralahzadch et al.) 2019).. we achieve the improvements of 3.94% EC and 6.27% NEWS. Compared with strong baseline, BiLSTM Partial CRF, the increases of FI scores are 9.20% and 8.21%. %. To make fair comparisons, we replace BERT with LSTM. Even so, we still outperform (Nooralahzadeh et al. (019)) by 1.76% EC and 2.52% NEWS. All these strongly confirm the effectiveness of our model.
6.5 RATIO À IN NEGATIVE SAMPLING
Intuitively, setting ratio À (see Sec. 4.2) as too large values too small values both are inappropri- ate. Large ratios increase the risks training negatives containing unlabeled entities. Small ratios reduce the number of negative instances used for training. leading to underfitting. Fig shows the experiments on some synthetic datasets with the ratio À of our method being 0.1, 0.2, ... 0.9. From that all the curves roughly arched, which verifies our intuition. Besides, it, find that 0.3 < < 0.4 performs well in all the cases.
6.6 VALIDITY OF DEGRADATION RATES
As Table shows, we use Pearson's 's correlation coefficient (PCC) to measure the statistical corre- lations between degradation rates (e.g., misguidance rate 3p)) and the Fl score. We can see that the correlation scores are generally close to 11 For example, for LSTM Tagging, datasets built from CoNLL-2003, the correlation score of erosion rate ap is -0.90 and that mis- guidance rate 3p is -0.96. The results indicate that not only degradation rates quantify specific impacts of unlabeled entities but also their negative values change synchronously with the Fl score. We conclude that degradation rates are appropriate metrics for evaluation.
the synthetic
hhaps://githuh.comaaaaajinieee incomplete annotation.
8

---

Published as a conference paper at ICLR 2021
7 RELATED WORK
NER is classical task in information extraction. Previous works commonly treat it as a sequence labeling problem by using IOB tagging scheme (Huang et al., 2015: Akbik et al. 2018 Luo et al, 2020 Li et al. c0 Each word in the sentence is labeled an entity, Ttt if it's inside but not the first one within the entity, or otherwise. This approach extensively studied in prior works. For example, Akbik al. (2018) propose Flair Embedding that pretrains character embedding in large corpora and uses it rather than token representations to represent a sentence. Recently, there is growing interest in span-based models (Li et al.) 2020a] Yu et al. 2020). They treat the spans, instead of single words, as the basic units for labeling. For example, Li
B B-tag if it the beginning of
(2020a) present BERT-MRC that regards NER as a MRC task, where named entities are extracted as retrieving answer spans. baaad models are also prevalent in language modeling (Li et al. 2020d), syntactic analysis (Stern et al., 2017). etc.
In some practical applications (e.g,, fine-grained NER (Zhang et al. (0200), NER models are faced with unlabeled entity problem, where the unlabeled entities seriously degrade the performances of models. Several approaches to this issue have been proposed. Fuzzy CRF and AutoNER (Shang) cct al. 2018b) allow learning from high-quality phrases. However, since these phrases are obtained through distant supervision, the unlabeled entities in the corpora may still be missed. PU learn- ing (Peng et al. 2019 Mayhew et al., 2019) unbiasedly and consistently estimates the training loss. Nevertheless, the unlabeled entities still impact the classifiers of the corresponding entity types and, importantly, the model can't 't disambiguate neighboring entities. Partial CRF (Yang et al.] [2018] Die et al. 2019) supports learning from incomplete annotations. However, because fully annotated cor- pora are still needed to training models with true negative instances, this type of approach is not applicable to the situations where no high-quality data is available.
8 CONCLUSION
In this work, we study what are the impacts of unlabeled entities on NER models and how to ef- fectively eliminate them. Through empirical studies performed on synthetic datasets, we find two causes: the reduction of annotated entities and treating unlabeled entities training negatives. The first cause has fewer influences than the second one and can be mitigated by adopting pretraining language models. The second cause seriously misleads the models in training and greatly affects their performances. Based on the above observations, we propose a novel method that is capable of eliminating the misguidance of unlabeled entities during training. The core idea is apply neg- ative sampling that avoids training NER models with unlabeled entities. Experiments on synthetic datasets and real-world datasets demonstrate that our model handles unlabeled entities well and sig- nificantly outperforms previous baselines. On well-annotated datasets, our model is competitive with the existing state-of-the-art approach.
REFERENCES
Alan Akbik, Duncan Blythe, and Roland Vollgraf. Contextual string embeddings for sequence labeling. In Proceedings of the 27th International Conference on Computational Linguistics, pp. 1649, 2018
Qian Chen, Xiaodan Zhu, Zhen Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguisties (Volume I: Long Pupers), pp. 1657-1668, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 1//53/11/P17-115/ URL https:
7/ f/www. www.ww.lwwbbo. org/ hnthoooyy/111 1152
Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pp. 4171-4186 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/N19-11 1423. URL https: 7/ 7/www. web. www....ogg/aaahooooy////// 9-

---

Published as a conference paper at ICLR 2021
Jason Fries, Sen Wu, Alex Ratner, and Christopher Ré. Swellshark: A generative model for biomed- ical named entity recognition without labeled data. arXiv preprint arXiv:1704.06360, 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Zhiheng Huang, Wei Xu, and Kai Yu. Bidirectional Istm-crf models for sequence tagging. arXiv preprint 1.Xiv:5508.015 2015.
Zhanming Jic, Pengjun Xie, Wei Lu, Ruixue Ding, and Linlin Li. Better modeling of incom- plcte annotations for named entity recognition. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume (Long and Short Papers), pp. 729-734, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10.18653/1//119-1077 URL https: www www.:w.wwwwb...... org www.oorgg/noooog//......
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:141266980 2014.
John Lafferty, Andrew McCallum, and Fernando CN Pereira. Conditional random fields: Proba- bilistic models for segmenting and labeling sequence data. 2001.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 260-270, San Diego, California, June 2016. Association for Computational /
Linguistics. doi: 10.18653/v1/N16- 1030. URI.https: //wp:://wwwwww www..www.......o.....oo.... a cl web.
N1 6 1116-1033 1030
Gina-Anne Levow. The third international chinese language processing bakcoff:: Word segmenta- tion and named entity recognition. In Proceedings of the Fifth SIGHAN Workshop Chinese Language Processing, pp. 108-117, 2006.
Xiao-Li Li and Bing Liu. Learning from positive and unlabeled examples with different data distri- butions. In European conference on machine learning, pp. 218 229. Springer, 2005.
Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. A unified MRC framework for named entity recognition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5849-5859, Online, July 2020a. Asso- ciation for Computational Linguistics. doi: 101186653112222..cc 511 URL https:
// /www.ww aclweb www.:ogggtaaa.eee... anthologg/h2200000 acl:0..cc111 maiimmmain... 519
Yangming Li, Han Li, Kaisheng Yao, and Xiaolong Li. Handling rare entities for neural sequence labeling. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics. pp. 6441-6451, Online, July 2020b. Association for Computational Linguistics. doi:
10. 12116633/1/000...clmmaaaaa
URL https:: //ww:: ac web. www.oorg//atthhclgg///000
acl Dcl-main.55 574
Yangming Li, Lemao Liu, and Shuming Shi. Segmenting natural language sentences via lexical unit analysis. arXiv preprint arXiv:2012.05418, 2020c.
Yangming Li, Kaisheng Yao, Libo Qin, Shuang Peng, Yijia Liu, and Xiaolong Li. Span-based neural buffer: Towards efficient and effective utilization of long-distance context for neural sequence models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8277- 8284, 2020d.
Xiao Ling and Daniel S Weld. Fine-grained entity recognition. In AAAI, volume 12, pp. 94-100, 2012.
Ying Luo, Fengshun Xiao, and Hai Zhao. Hierarchical contextualized representation for named entity recognition. In AAAI, pp. 8441-8448, 2020.
10

---

Published as a conference paper at ICLR 2021
Xuezhe Ma and Eduard Hovy. End-to-end sequence labeling via bi-directional LSTM-CCNN-CRRE In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1064-1074, Berlin, Genmany, August 2016. Association for Com- putational Linguistics. doi: 10. 10.186653/1//11-1110.
URL. https:/ / www. ac web. org/
anthol oothhloogy/P1-110011
Stephen Mayhew, Snigdha Chaturvedi, Chen-Tse Tsai, and Dan Roth. Named entity recogni- tion with partially annotated training data. In Proceedings of the 23rd Conference on Com- putational Natural Language Learning (CoNLL), pp. 645-655, Hong Kong, China, Novem- ber 2019. Association for Computational Linguistics. doi: 10. 1118653/11/K1--106)) URL
https:/ www..cclwebe.. org/ wtt.a/atoooooooooooooooooo htaag//aaahooooo/111-00000 (11-10000199100
G. Mesnil, Y. Dauphin, K. Yao, Y. Bengio, L.. Deng, D. Hakkani-Tur, X. He, L. Heck, G. Tur, D. Yu, and G. Zweig. Using recurrent neural networks for slot filling in spoken language understanding. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 23(3):530-539 2015, doi: 10.1109/TASLLP22444333666
Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pp. 1003-1011, 2009.
Farhad Nooralahzadeh, Jan Tore Lønning, and Lilja Øvrelid. Reinforcement-based denoising of distantly supervised ner with partial annotation. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pp. 225 225-233, 2019.
Minlong Peng, Xiaoyu Xing, Qi Zhang, Jinlan Fu, and Xuanjing Huang. Distantly supervised named entity recognition using positive-unlabeldd learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2409-240224 2419, Florence, Italy, July 2019. Association for Computational Linguisties. doi: 10. 10.18653/v 1/119-123. URL https:
7/ www. acwwe....wb... htt.://..rr..aatoooo/o//ssee
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1555 1543, 2014.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. Towards robust linguistic analysis using ontonotes. In Proceedings of the Seventeenth Conference Computational Natural Language Learning, pp- 143-152, 2013.
Libo Qin. Wanxiang Che, Yangming Li, Haoyang Wen, and Ting Liu. A stack- propagation frame- work with token-level intent detection for spoken language understanding. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th In- pp. 2078-
ternational Joint Conference on Natural Language Processing 2087, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 1214
18. 1214, URL httph:ttt//// www acl www.wwe.boooo http://.w..gggttttooo/o///.aa//.
Xiang Ren, Ahmed El-Kishky, Chi Wang, Fangbo Tao, Clare R Voss, and Jiawei Han. Clustype: Effective entity recognition and typing by relation parase-based clustering. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp- 995-1004, 2015.
Erik Sang and Fien De Meulder. Introduction to the con1l-2003 shared task: Language- independent named entity recognition. arXiv preprint cs/0306050, 2003.
Jingbo Shang, Jialu Liu, Meng Jiang, Xiang Ren, Clare R Voss, and Jiawei Han. Automated phrase mining from massive text corpora. IEEE Transactions on Knowledge and Data Engineering, 30 (105:1825-1877, 2018a.
11

---

Published as a conference paper at ICLR 2021
Jingbo Shang, Liyuan Liu, Xiaotao Gu, Xiang Ren, Teng Ren, and Jiawei Han. Learning named entity tagger using Semain-specific dictionary. In Proceedings of the 2018 Conference on Em- pirical Methods in Natural Language Processing, pp. 2054-2064, Brussels, Belgium, October- November 201 Association for Computational Linguistics. doi: 10.8////11/D11812230 URL https:: wwwp://www... ac web. o rg/ www.oooogo..oooooooooooooo wgy.
Yan Song. Shuming Shi, Jing Li, and Haisong Zhang. Directional skip-gram: Explicitly dis- tinguishing left and right context for word embeddings. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Association for Computational Linguisties: Hu- man Language Technologies, Volume (Short Papers), pp- 175-180, New Orleans, Louisiana, June 2018, Association for Computational Linguistics. doi: 10.18653/VI/N18- 2028. URL 2028
https: http://wwwwwwww../o//aoaooaaaaa.
Mitchell Stern, Jacob Andreas, and Dan Klein. A minimal span-based neural constituency parser. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume
1: Long Papers), pp. 818-827, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 1/177-10//11 htLhttps:: www wwwweb. org/ ort.ooooogggggy/////
P17-1075
Yuta Tsuboi, Hisashi Kashima, Shinsuke Mori, Hiroki Oda, and Yuji Matsumoto. Training con- ditional random fields using incomplete annotations. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pp. 897-904, 2008
Yaosheng Yang, Wenliang Chen, Zhenghua Li, Zhengqiu He, and Min Zhang. Distantly super- vised ner with partial annotation learning and reinforcement learning. In Proceedings of the 27th International Conference on Computational Linguistics, pp. 2159-2169, 2018
Juntao Yu, Bernd Bohnet, and Massimo Poesio. Named entity recognition dependency parsing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. Online, July 2020. Association for Computational Linguistics. doi: 18653vv// 2020.acl- ma7.-mmm...777 URL https: www.:://wwwwwwww................ wtt.://wwww...re//ooooooooo/////
577
Haisong Zhang, Lemao Liu, Haiyun Jiang, Yangming Li, Enbo Zhao, Kun Xu, Linfeng Song, Sun- cong Zheng, Botong Zhou, Jianchen Zhu, et al. Texsmart: text understanding system for fine-grained ner and enhanced semantic analysis. arXiv preprint 2rXiv:20/2.156:: 2020.
12


Reviews:
[{"review_id": "wM0Cz3H5hlb", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates the unlabeled entity problem, which is generally observed in the manual annotation setting and distant supervision as well. The unlabeled problem is important and some existing works focus on solving the problems using partial CRF setting or data selector. The main observation of this paper lies in two aspects: 1) comparison between the reduction of annotated entities or treating unlabeled entities as negative instances.  Most interestingly, the authors show the observed difference between pre-trained language models and LSTM-based models.  Based on the observations, they propose a general approach to eliminate the misguidance brought by unlabeled entities and such a simple design shows good performances.  \n\nThe Paper is overall well written and easy to follow. But  I still have a few questions and want to get answers from authors. \n\nQuestions:\n\n1) The first question is about 4.2 Training via Negative sampling on page 5. I am not quite sure about the procedure. Negative instance candidates are randomly selected from original sentences.  You use \\hat{y}, which is a subset of randomly selected span to replace a missed entity set defined in Eq. (2)?\n\n2) Could you expand more about Equation 8 to add more details?\n\n3) The unlabeled entity problem is most serious in the distant supervision setting. However, the distant supervision setting suffers from entity ambiguation and unlabeled entity problem simultaneously.  How do you think your design to tackle entity ambiguation problem? Moreover, in the distant supervision experiment in Table  3, how will you model compare with other distant supervision models like AutoNER?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "dN5WewHl75m", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This article deals with the problem of partially labeled dataset: if some entities are missing, how SOTA approaches are going to behave? To answer this question, the authors degrade CoNLL classical dataset by masking a pourcentage of the labeled data. Then, they wonder which part of the missing performance is due to the lack of labels and which part is due to the incorrect labelling of discarded supervision.\nThe experiments are well explained and interesting on synthetic dataset. Then the authors propose a new cross entropy loss to test their hypothesis on real data by sampling high confidense negative samples as ground truth. It is a way of performing distillation on the model using negative sampling.\n\nConsistant & relevant work that deserves to be publised in ICLR.\n\n* We wonder what would give a classical distillation process on this task. [even if relevant comparison are made with results from the litterature]\n\n* Given the architecture, we wonder what is the detailed learning procedure: it seems clear that the network is first trained on the real ground truth and then refined using the distillation loss. Section 3.1 is a little bit short on this point.\n\n* Regarding the model, equation (4) is not discussed nor analysed.\n\n* The approach is rather simple but elegant.", "extra_scores": {}, "extra_text": {}}, {"review_id": "JZDh9tCbsgS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper conducted an empirical analysis on the unlabeled entity problem in the NER task. It concluded that there are two reasons to affect the NER model's performance:  the reduction of annotated entities, and treating unlabeled entities as negative instances. Experiments showed that the latter reason gave a much more negative impact on the NER models. \n\nThis empirical study was conducted on the synthetic datasets which were extracted from two English NER datasets using negative sampling. The way of constructing synthetic datasets should be explained clearly (Section 3.1). It is not clear how does the negative sampling applied on the dataset (in sentence level? entity level? token level?). ", "extra_scores": {}, "extra_text": {}}, {"review_id": "3E-03klkV0", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on the unlabeled entity problem in NER, where the entities of a sentence are incomplete annotated. Since some entities may not be annotated, the performance of models can be degraded. This paper analyzes the performance degradation by evaluating synthetic datasets and finds that all the unlabeled entities are treated as negative instances is the main factor of the performance degradation. To alleviate the performance degradation, this paper proposes a negative sampling approach that considers only a small subset of unlabeled entities in order to reduce the impacts of unlabeled entities. The experimental results show that the proposed method achieves better performances compared to previous studies on real-world datasets and achieves competitive performances compared to the state-of-the-art methods on well-annotated datasets.\n\nStrong points:\n+ The analysis using two proposed metrics to find causes of the performance degradation of the unlabeled entity problem is quite interesting.\n+ The idea of using the negative sampling under the data imbalance situation, especially in NER, seems reasonable.\n+ The paper is well-written and easy to follow. It provides comprehensive experiments on several datasets.\n\nConcerns: \n1. The key concern about this paper is the missing of some references. There are some previous studies related to the unlabeled entity problem. Particularity, this problem is quite related to or almost the same as the “Incomplete annotations” at [1] or “Partially annotated training data” at [2]. In addition, the unlabeled entity problem has a little relevance to the data imbalances [3] if we assume that all unlabeled spans are negative instances. Therefore, the proposed method should be compared with these previous studies experimentally and/or theoretically.\n[1] Jie et al., Better Modeling of Incomplete Annotations for Named Entity Recognition, NAACL 2019.\n[2] Mayhew et al., Named Entity Recognition with Partially Annotated Training Data, CONLL 2019.\n[3] Li et al., Dice Loss for Data-imbalanced NLP Tasks, ACL 2020.\n2. In the experiments on the real-word dataset, the proposed method only compared with weak baselines. The reviewer suggests the authors to evaluate with the state-of-the-art NER methods such as BERT-MRC or BERT-Biaffine Model.\n3. At the inference time, every span should be input to the MLP layer to obtain the predicted score. That is, the proposed method takes more time (O(n^2)) to infer new sentences compared to other methods (O(n)). Furthermore, this paper uses heuristic(s) when the spans of inferred entities intersects. However, there is no discussion about them.\n\nMinor comments:\n1. In the section 3.2, the figures and the contents about the figures are inconsistent. That is, the middle parts of the figures are for the change in misguidance rates and the right-hand ones are for the change of the erosion rates.\n2. For batching with sampling, it would be better to provide more details about it.\n3. It would be nice to show a model convergence or loss convergence graph because this paper uses random sampling.\n4. It would be better to show the correlation in numbers between the f1 scores and the proposed rates.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the unlabeled entity problem in NER. Specifically, performance degradation in training of NER models due to unlabeled entities. It analyzes the reason through evaluation on synthetic datasets and finds that it is due to the fact that all the unlabeled entities are treated as negative examples. To cope with the problem, it proposes a negative sampling method which considers the use of only a small subset of unlabeled entities. Experimental results show that the proposed method achieves better performances than the baselines on real-world datasets and achieves competitive performances compared with the state-of-the-art methods on well-annotated datasets.

Pros
•	The paper is clearly written.
•	The proposed method appears to be technically sound.
•	Experimental results support the main claims.
•	The findings in the paper are useful for the field.

Cons
•	Novelty of the work might not be enough.

The authors have addressed some clarity and reference issues pointed out by the reviewers in the rebuttal.  Discussions have been made among the reviewers.

Decision: Accept (Poster)