Venue: NAACL 2019

# Better Modeling of Incomplete Annotations for Named Entity Recognition

Zhanming Jie\({}^{1}\), Pengjun Xie\({}^{2}\), Wei Lu\({}^{1}\), Ruixue Ding\({}^{2}\) and Linlin Li\({}^{2}\)

\({}^{1}\)StatNLP Research Group, Singapore University of Technology and Design

\({}^{2}\)DAMO Academy, Alibaba Group

zhanming_jie@mymail.sutd.edu.sg, luwei@sutd.edu.sg {chengchen.xpj, ada.drx, linyan.lll}@alibaba-inc.com

###### Abstract

Supervised approaches to named entity recognition (NER) are largely developed based on the assumption that the training data is fully annotated with named entity information. However, in practice, annotated data can often be imperfect with one typical issue being the training data may contain incomplete annotations. We highlight several pitfalls associated with learning under such a setup in the context of NER and identify limitations associated with existing approaches, proposing a novel yet easy-to-implement approach for recognizing named entities with incomplete data annotations. We demonstrate the effectiveness of our approach through extensive experiments.1

Footnote 1: Our code and data are available at [http://statnlp.org/research/ie](http://statnlp.org/research/ie).

## 1 Introduction

Named entity recognition (NER) Tjong Kim Sang (2002); Tjong Kim Sang and De Meulder (2003) as one of the most fundamental tasks within natural language processing (NLP) has received significant attention. Most existing approaches to NER focused on a supervised setup, where fully annotated named entity information is assumed to be available during the training phase. However, in practice, obtaining high-quality annotations can be a very laborious and expensive process Snow et al. (2008). One of the common issues with data annotations is there may be incomplete annotations.

Figure 1 shows an example sentence with two named entities "_John Lloyd Jones_" and "_BBC radio_" of type per (person) and org (organization), respectively. Following the standard bioies tagging scheme Ramshaw and Marcus (1999); Ratinov and Roth (2009), the corresponding gold label sequence is shown below the sentence. When the data annotations are incomplete, certain labels may be missing from the label sequence. Properly defining the task is important, and we argue there are two possible potential pitfalls associated with modeling incomplete annotations, especially for the NER task.

Several previous approaches assume the incomplete annotations can be obtained by simply removing either word-level labels Fernandes and Brefeld (2011) or span-level labels Carlson et al. (2009). As shown in Figure 1, under both assumptions (i.e., _A.1_ and _A.2_), there will be words annotated with o labels. The former approach may even lead to sub-entity level annotations (e.g., "_radio_" is annotated as part of an entity). However, we argue such assumptions can be largely unrealistic. In practice, annotators are typically instructed to annotate named entities for complete word spans only Settles et al. (2008); Surdeanu et al. (2010). Thus, sub-entity level annotations or o labels 2 should not be assumed to be avail

Figure 1: An example sentence with gold named entity annotations and different assumptions (i.e., _A.1_ to _A.3_) on _available labels_. “-” represents a missing label.



able (_A.3_). Therefore such approaches are making sub-optimal assumptions on the _available labels_.

When the proper assumptions on the available labels are made, one can typically model the missing labels as latent variables and train a latent-variable conditional random fields model Quattoni et al. (2005). One such approach is presented in Bellare and McCallum (2007). Their work focused on the citation parsing3 (i.e., sequence labeling) task which does not suffer from the above issue as no o label is involved. However, though the approach was shown effective in the citation parsing task, we found its effectiveness does not transfer to the NER task even in the absence of the above available labels issue. As we would highlight later, the reason is related to the undesirable assumptions on the _unavailable labels_.

Footnote 3: The task is to tag the BibTex records with different labels (i.e., “title”, “author”, “affiliation” and so on).

In this work, we tackle the incomplete annotation problem when building an NER system, under a more realistic yet more challenging scenario. We present a novel, effective, yet easy-to-implement approach, and conduct extensive experiments on various datasets and show our approach significantly outperforms several previous approaches.

## 2 Related Work

Previous research efforts on partially annotated data are mostly based on the conditional random fields (CRF) Lafferty et al. (2001), structured perceptron Collins (2002) and maxmargin Tsochantaridis et al. (2005) (e.g. structural support vector machine) models. Bellare and McCallum (2007) proposed a missing label linear-chain CRF4 which is essentially a latent-variable CRF Quattoni et al. (2005) on citation parsing McCallum et al. (2000). This model had also been used in part-of-speeching tagging and segmentation task with incomplete annotations Tsuboi et al. (2008); Liu et al. (2014); Yang and Vozila (2014). Yang et al. (2018) showed the effectiveness of such a model on Chinese NER with incomplete annotations due to the fact that they required a certain number of fully annotated data to perform joint training. Greenberg et al. (2018) applied this model on a biomedical NER task and achieved promising performance with incomplete annotations. However, in their assumption for the incomplete annotations, the O labels are still considered, which we believe is not realistic. Carlson et al. (2009) modified the structured perceptron algorithm and defined features only on the tokens with annotated labels in partially labeled sequences. Fernandes and Brefeld (2011) and Lou et al. (2012) proposed to use a large-margin learning framework similar to structured support vector machines with latent variables Yu and Joachims (2009).

Footnote 4: This model was also named as Partial CRF Carlson et al. (2009) and EM Marginal CRF Greenberg et al. (2018).

## 3 Approach

Given the input word sequence \({\bf x}\), the NER task is to predict a label sequence \({\bf y}\) that encodes the NER information (e.g., in a form following the bioes tagging scheme). Given a training set that consists of completely labeled data \(\mathcal{D}\), one can tackle this problem using a standard linear-chain conditional random field (CRF) Lafferty et al. (2001) whose loss function is as follows:5

Footnote 5: In practice, we also have an \(L_{2}\) regularization term, which we exclude from the formula for brevity.

\[\mathcal{L}({\bf w})=-\sum_{i}\log p_{\mathbf{w}}({\bf y}^{(i)}|{\bf x}^{(i)}) \tag{1}\]

Figure 3: Graphical illustrations on different assumptions on _unavailable labels_, where the entity “_John Lloyd Jones_” of type per is labeled but “_BBC radio_” of type org is missing. Each path refers to one possible complete label sequence, and the density of the color indicates probability (we excluded b and e tags for brevity).

 

where \(({\bf x}^{(i)},{\bf y}^{(i)})\) is the \(i\)-th instance from \(\mathcal{D}\).

Now, assume we have an incomplete label sequence \({\bf y}_{p}^{(i)}\). From such a \({\bf y}_{p}^{(i)}\) we should be able to derive a set of all possible complete label sequences that are compatible with (i.e., contain) the incomplete label sequence, and let us call this set \(\mathcal{C}({\bf y}_{p}^{(i)})\). We can rewrite the above function as:

We illustrate in Figure 3 several previous approaches as well as our approach. In this example, the entity _BBC radio_ of type org is not annotated. Figure 3 shows a single path that corresponds to the gold label sequence. Figure 3 illustrates a naive approach, where we regard all the missing labels as o labels. This essentially assumes that the \(q\) distribution in the above equation puts all probability mass to this single label sequence, which is an incorrect assumption.

Now let us look at what assumptions on \(q\) have been made by the existing approach of Bellare and McCallum (2007). The model regards the missing labels as latent variables and learns a latent variable CRF using the following loss:

\[-\sum_{i}\log\sum_{{\bf y}\in\mathcal{C}({\bf y}_{p}^{(i)})}p_{{\bf w}}({\bf y }|{\bf x}^{(i)}) \tag{2}\]

The resulting model is called _missing label linear-chain CRF_ (_M-CRF_) 6. As we can see from the above function, this is essentially equivalent to say \(q\) is a uniform distribution that assigns equal probabilities to all possible complete label sequences in \(\mathcal{C}({\bf y}_{p}^{(i)})\).

Footnote 6: Similar assumptions have also been made by Carlson et al., 2009; Fernandes and Brefeld, 2011), but they used structured perceptron Collins, 2002 instead.

We believe such an assumption on \(q\) that describes _unavailable labels_ can be improved. As we can see from the above example in Figure 3, a more desirable assumption about \(q\) is to put more probability mass to a path that is close to the gold path. In practice, their approach worked for the task of citation parsing, where the \(q\) distribution may not deviate much from the uniform distribution (Figure 3) in such a task. However, in the task of NER, we find such a simple treatment to the \(q\) distribution often leads to sub-optimal results (as we can see in the experiments later) as the \(q\) distribution is highly skewed due to the large amount of o labels. This observation motivates us to find a proper way to define \(q\) that can approximate the gold label distribution in this work.

### Estimating \(q\)

Inspired by the _classifier stacking_ technique used in Nivre and McDonald (2008), we empirically found that a reasonable \(q\) distribution can be acquired in a \(k\)-fold cross-validation fashion.

We first start with an initialization step where we assign specific labels to words without labels, forming complete label sequences (we will discuss our initialization strategy in experiments). Next, we perform \(k\)-fold cross-validation on the training set. Specifically, each time we train a model with (\(k\)-1) folds of the data and based on the learned model we define our \(q\) distribution.

We describe two different ways of defining the \(q\) distribution, namely the _hard_ approach, and the _soft_ approach. In the _hard_ approach, the resulting \(q\) distribution is a collapsed distribution that assigns probability 1 to a single complete label sequence, whereas in the _soft_ approach each possible label sequence will get a certain probability score.

In the _hard_ approach, after training a model from (\(k\)-1) folds, we apply a constrained Viterbi procedure 7 to the sentences in the remaining fold. In the _soft_ approach, we use a constrained version of the forward-backward procedure and calculate the marginal probabilities associated with each label at each unlabeled position. The score of each complete label sequence can then be calculated as a product of all such marginal probabilities. We note that in the above procedure the estimation to \(q\) depends on the initialization. Thus we iterate the above procedure, which allows us to converge to an improved \(q\).

Footnote 7: The algorithm will ensure the resulting complete label sequence is compatible with the incomplete label sequence.

## 4 Experiments

We conduct experiments on two standard NER datasets - CoNLL-2003 English and CoNLL-2002 Spanish datasets that consist of news articles. We

\begin{table}
\begin{tabular}{l|c c|c c|c c|c} \hline Dataset & Training & Validation & Test & Earties \\  & reentry & test & reentry & test & reentry & fremt & \# (v) \\ \hline CoNLL-2003 & 23.499 & 14,041 & 5.942 & 3.250 & 5.648 & 3.453 & 4 & 2.36 \\ CoNLL-2002 & 18.796 & 3.822 & 4.338 & 1.914 & 3.559 & 1.516 & + 12.4 \\ Taubuo & 29.397 & 6.000 & 4.941 & 998 & 4.866 & 1.900 & 4 & 51.0 \\ Youtsu & 12.754 & 8.001 & 1.580 & 1.000 & 1.570 & 1.001 & 3 & 41.7 \\ \hline \end{tabular}
\end{table}
Table 1: Data statistics for the datasets.

 

notice that incomplete annotation issue is very common in the industry setup. Therefore we also consider two new datasets from industry - Taobao and Youku datasets8 consisting of product and video titles in Chinese. We crawled and manually annotated such data with named entities9. Table 1 shows the statistics of the datasets. The last two columns show the number of entity types and the percentage of words (i.e., \(c\) in Table 1) that are parts of an NE. Based on our assumption on the _available labels_ in Section 1, we randomly remove a certain number of entities as well as all o labels and use \(\rho\) to represent the ratio of annotated entities. For example, \(\rho=0.6\) means we keep 60% of all the entities and remove the annotations of 40% of the entities. Meanwhile, the o labels are considered unavailable.

Footnote 7: [http://www.taobao.com/](http://www.taobao.com/) and [http://www.youku.com/](http://www.youku.com/)

We follow Lample et al. (2016) and apply the bidirectional long short-term memory Hochreiter and Schmidhuber (1997) (BiLSTM) networks as the neural architecture for all baselines and our approaches. Specifically, we implement the following baselines: a _Simple_ model which is a linear-chain LSTM-CRF model and we treat all missing labels as o; the missing label CRF Bellare and McCallum (2007); Greenberg et al. (2018) (LSTM-M-CRF) model; the _partial perceptron_Carlson et al. (2009) model, which is a structured perceptron Collins (2002) but only considers the scores on the words with available labels; the _transductive perceptron_Fernandes and Brefeld (2011) model where they introduce a Hamming loss function during the perceptron training process; lastly, we train an LSTM-CRF Lample et al. (2016) with complete annotations as the upper bound (_Complete_). For English and Spanish, we use exactly the same embeddings used in Lample et al. (2016). We train our Chinese character embeddings on the Chinese Gigaword10 corpus. The resulting implementation achieves 90.9 and 85.8 \(F\)-scores on CoNLL-2003 English and CoNLL-2002 Spanish datasets, respectively. These benchmark results are comparable with the results reported in the state-of-the-art NER systems Lample et al. (2016); Ma and Hovy (2016); Reimers and Gurevych (2017).

Footnote 10: Similar to the EM procedure, a good initialization is crucial for our approach. We found using random initialization can lead to substantially worse results and a better initialization can be used to further improve the results.

For initialization in our approaches, we run the _Simple_ model on each fold and use the results to initialize our \(q\) distribution11. Detailed descriptions on experiment settings (e.g., hidden dimension of LSTM and optimizer) and baseline systems are provided in supplementary material.

Footnote 11: Similar to the EM procedure, a good initialization is crucial for our approach. We found using random initialization can lead to substantially worse results and a better initialization can be used to further improve the results.

Main ResultsTable 2 presents the comparisons among all approaches on four datasets with \(\rho=0.5\) and \(k=2\). Our preliminary experiments show that a larger \(k\) value have a negligible effect on the results. A similar finding was also reported in Nivre and McDonald (2008). The _Simple_ model has high precision and low recall as it treats unknown labels as o. Previous models for incomplete annotations achieve a much lower \(F\)-score compared to the _Simple_ model and our approaches. Due to their uniform assumption on \(q\) over the missing labels, these models typically can recall more entities. The partial perceptron Carlson et al. (2009) among these three models yields a relatively lower recall as features are not defined over the words wit

\begin{table}
\begin{tabular}{l|r r r|r r r|r r r|r r r} \hline \multirow{2}{*}{**Approach**} & \multicolumn{3}{c|}{CoNLL-2003} & \multicolumn{3}{c|}{CoNLL-2002} & \multicolumn{3}{c|}{Taobao} & \multicolumn{3}{c}{Youku} \\  & \(P\). & \(R\). & \(F\). & \(P\). & \(R\). & \(F\). & \(P\). & \(R\). & \(F\). & \(P\). & \(R\). & \(F\). \\ \hline _Simple_ & 93.6 & 68.6 & 79.2 & 86.8 & 57.0 & 68.8 & 83.1 & 46.7 & 59.8 & 91.1 & 49.1 & 63.8 \\ LSTM-M-CRF & 13.0 & 90.5 & 22.8 & 6.2 & 84.5 & 11.6 & 33.0 & 83.0 & 47.2 & 17.6 & 83.7 & 29.1 \\ LSTM-Partial Perceptron & 27.6 & 82.0 & 41.3 & 22.3 & 66.3 & 33.3 & 26.6 & 59.7 & 36.8 & 19.7 & 69.0 & 30.6 \\ LSTM-Transductive Perceptron & 11.7 & 90.5 & 20.7 & 6.1 & 84.3 & 11.4 & 32.9 & 82.2 & 47.0 & 16.0 & 81.9 & 26.7 \\ \hline Ours (_hard_) & 88.1 & 89.9 & 89.0 & 80.8 & 82.1 & 81.5 & 69.3 & 77.4 & 73.1 & 77.2 & 79.8 & 78.5 \\ Ours (_soft_) & 89.0 & 90.1 & **89.5** & 81.3 & 82.7 & **82.0** & 69.7 & 78.1 & **73.7** & 78.1 & 79.6 & **78.8** \\ \hline _Complete_ & 91.0 & 90.8 & 90.9 & 85.7 & 85.8 & 85.8 & 82.3 & 82.6 & 82.4 & 83.0 & 81.7 & 82.4 \\ \hline \end{tabular}
\end{table}
Table 2: Performance comparison between different baseline models and our approaches on 4 datasets with \(\rho=0.5\) (for _Complete_ model, \(\rho=1.0\)).

 

The difference in \(F\)-score between these three models and the _Simple_ model is more significant on the two CoNLL datasets than on Taobao and Youku. As shown in Table 1, the latter two datasets have more words labeled as parts of entities (i.e., a higher \(c\)). This means these industrial datasets have less o labels, making such baseline models suffer less from their assumptions on the _unavailable labels_. With a properly learned \(q\) distribution, our approaches improves the recall score over the _Simple_ model while preserving a high precision. Our _soft_ approach consistently achieves a better \(F\)-score compared with the _hard_ approach on all datasets with \(p<0.001\). Compared to the _Complete_ upper bound, our _soft_ approach are still more than 3% lower in \(F\)-score on the CoNLL-2002, Taobao and Youku datasets. However, we can see that the _soft_ approach achieves much higher performance compared to this variant on other datasets. We attribute this phenomenon to our approaches' ability in retrieving most of the entities in the training set. Empirically, we found our _soft_ approach can recover 94% of the entities in the training set of the CoNLL-2003 dataset.

The overall results show the underlying scenario is challenging for commonly adopted models in handling incomplete annotations and our approaches can achieve better performance compared with them.

Effect of \(\rho\)We conduct experiments with different \(\rho\) from \(0.1\) to \(0.9\) for our _soft_ approach against the _Simple_ and LSTM-M-CRF models. Figure 4 shows how the precision, recall and \(F\)-score on CoNLL-2003 change as we increase \(\rho\). The \(F\)-score of the _Simple_ baseline increases progressively as \(\rho\) increases. LSTM-M-CRF always maintains a low \(F\)-score which is not sensitive to different \(\rho\) values because of their high recall and low precision values as we can see in Figure 4 (a, b). The improvement of our approach attributes to the increase of recall as the precision is constantly high and stable. We can see that our _soft_ approach performs particularly well when \(\rho\) is larger than 0.3 which indicates a modest amount of missing labels in practice.

## 5 Conclusions and Future Work

In this work, we identified several limitations associated with previous assumptions when performing sequence labeling with incomplete annotations, and focused on the named entity recognition task. We presented a novel and easy-to-implement solution that works under a realistic and challenging assumption on the incomplete annotations. Through extensive experiments and analysis, we demonstrated the effectiveness of our approach.

Although we focused on the task of named entity recognition in this work, we believe the proposed approach may find applications in some other sequence labeling tasks or other more general structured prediction problems where the issue of incomplete annotations is involved. We leave them as future work.

## Acknowledgments

We would like to thank the anonymous reviewers for their constructive comments on this work. This work is done under a collaborative agreement between SUTD and Alibaba on an Alibaba Innovative Research (AIR) Program funded by Alibaba, where Alibaba provided data and helped with experiments. We appreciate Alibaba's generosity in the agreement that makes it possible for us to make all data and code in this research publicly available upon acceptance of this paper. This work is also partially supported by SUTD project PIE-SGP-AI-2018-01.

Figure 4: Precision, Recall and \(F\)-score with different \(\rho\) on CoNLL-2003 dataset.

 



## References

* Bellare and McCallum (2007) Kedar Bellare and Andrew McCallum. 2007. Learning extractors from unlabeled text using relevant databases. In _Proceedings of International Workshop on Information Integration on the Web_.
* Carlson et al. (2009) Andrew Carlson, Scott Gaffney, and Flavian Vasile. 2009. Learning a named entity tagger from gazetteers with the partial perceptron. In _Proceedings of AAAI Spring Symposium: Learning by Reading and Learning to Read_.
* Collins (2002) Michael Collins. 2002. Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms. In _Proceedings of EMNLP_.
* Fernandes and Brefeld (2011) Eraldo R Fernandes and Ulf Brefeld. 2011. Learning from partially annotated sequences. In _Proceedings of ECML-KDD_.
* Greenberg et al. (2018) Nathan Greenberg, Trapit Bansal, Patrick Verga, and Andrew McCallum. 2018. Marginal likelihood training of bilstm-crf for biomedical named entity recognition from disjoint label sets. In _Proceedings of EMNLP_.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_, 9(8):1735-1780.
* Lafferty et al. (2001) John Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In _Proceedings of ICML_.
* Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In _Proceedings of NAACL_.
* Liu et al. (2014) Yijia Liu, Yue Zhang, Wanxiang Che, Ting Liu, and Fan Wu. 2014. Domain adaptation for crf-based chinese word segmentation using free annotations. In _Proceedings of EMNLP_.
* Lou et al. (2012) Xinghua Lou, Fred A Hamprecht, and IWR HCI. 2012. Structured learning from partial annotations. In _Proceedings of ICML_.
* Ma and Hovy (2016) Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional lstm-cnns-crf. In _Proceedings of ACL_.
* McCallum et al. (2000) Andrew Kachites McCallum, Kamal Nigam, Jason Rennie, and Kristie Seymore. 2000. Automating the construction of internet portals with machine learning. _Information Retrieval_, 3(2):127-163.
* Nivre and McDonald (2008) Joakim Nivre and Ryan McDonald. 2008. Integrating graph-based and transition-based dependency parsers. In _Proceedings of ACL_.
* Quattoni et al. (2005) Ariadna Quattoni, Michael Collins, and Trevor Darrell. 2005. Conditional random fields for object recognition. In _Proceedings of NIPS_.
* Ramshaw and Marcus (1999) Lance A Ramshaw and Mitchell P Marcus. 1999. Text chunking using transformation-based learning. In _Proceedings of Third Workshop on Very Large Corpora_.
* Ratinov and Roth (2009) Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In _Proceedings of CoNLL_.
* Reimers and Gurevych (2017) Nils Reimers and Iryna Gurevych. 2017. Reporting score distributions makes a difference: Performance study of lstm-networks for sequence tagging. In _Proceedings of EMNLP_.
* Settles et al. (2008) Burr Settles, Mark Craven, and Lewis Friedland. 2008. Active learning with real annotation costs. In _Proceedings of NIPS Workshop on Cost-Sensitive Learning_.
* Snow et al. (2008) Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y Ng. 2008. Cheap and fast--but is it good?: evaluating non-expert annotations for natural language tasks. In _Proceedings of EMNLP_.
* Surdeanu et al. (2010) Mihai Surdeanu, Remesh Nallapati, and Christopher Manning. 2010. Legal claim identification: Information extraction with hierarchically labeled data. In _Proceedings of LREC_.
* Sang (2002) Erik F Tjong Kim Sang. 2002. Introduction to the conll-2002 shared task: Language-independent named entity recognition. In _Proceedings of CoNLL_.
* Sang and De Meulder (2003) Erik F Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In _Proceedings of CoNLL_.
* Tsochantaridis et al. (2005) Ioannis Tsochantaridis, Thorsten Joachims, Thomas Hofmann, and Yasemin Altun. 2005. Large margin methods for structured and interdependent output variables. _Journal of machine learning research_, 6:1453-1484.
* Tsuboi et al. (2008) Yuta Tsuboi, Hisashi Kashima, Shinsuke Mori, Hiroki Oda, and Yuji Matsumoto. 2008. Training conditional random fields using incomplete annotations. In _Proceedings of COLING_.
* Yang and Vozila (2014) Fan Yang and Paul Vozila. 2014. Semi-supervised chinese word segmentation using partial-label learning with conditional random fields. In _Proceedings of EMNLP_.
* Yang et al. (2018) Yaosheng Yang, Wenliang Chen, Zhenghua Li, Zhengu Qiu He, and Min Zhang. 2018. Distantly supervised ner with partial annotation learning and reinforcement learning. In _Proceedings of COLING_.
* Yu and Joachims (2009) Chun-Nam John Yu and Thorsten Joachims. 2009. Learning structural svms with latent variables. In _Proceedings of ICML_