Venue: EMNLP 2020

# Coarse-to-Fine Pre-training for Named Entity Recognition

Mengge Xue\({}^{1,2}\) Bowen Yu\({}^{1,2}\) Zhenyu Zhang\({}^{1,2}\) Tingwen Liu\({}^{1,2*}\) Yue Zhang\({}^{3}\) and Bin Wang\({}^{4}\)

\({}^{1}\)Institute of Information Engineering, Chinese Academy of Sciences. Beijing, China

\({}^{2}\)School of Cyber Security, University of Chinese Academy of Sciences. Beijing, China

\({}^{3}\)School of Engineering, Westlake University, Hangzhou, China

\({}^{4}\)Xiaomi AI Lab, Xiaomi Inc., Beijing, China {xuemenge, yubowen, zhangzhenyu1996, liutingwen}@iie.ac.cn

yue.zhang@wias.org.cn wangbin11@xiaomi.cn

###### Abstract

More recently, Named Entity Recognition has achieved great advances aided by pre-training approaches such as BERT. However, current pre-training techniques focus on building language modeling objectives to learn a general representation, ignoring the named entity-related knowledge. To this end, we propose a NER-specific pre-training framework to inject coarse-to-fine automatically mined entity knowledge into pre-trained models. Specifically, we first warm-up the model via an entity span identification task by training it with Wikipedia anchors, which can be deemed as general-typed entities. Then we leverage the gazetteer-based distant supervision strategy to train the model extract coarse-grained typed entities. Finally, we devise a self-supervised auxiliary task to mine the fine-grained named entity knowledge via clustering. Empirical studies on three public NER datasets demonstrate that our framework achieves significant improvements against several pre-trained baselines, establishing the new state-of-the-art performance on three benchmarks. Besides, we show that our framework gains promising results without using human-labeled training data, demonstrating its effectiveness in label-few and low-resource scenarios.1

Footnote 1: The source code can be obtained from [https://github.com/strawberrys/CoFEE](https://github.com/strawberrys/CoFEE)

+
Footnote †: Corresponding Author

## 1 Introduction

Named Entity Recognition (NER) is the task of discovering information entities and identifying their corresponding categories, such as mentions of people, organizations, locations, temporal and numeric expressions Freitag (2004). It is an essential component in many applications including machine translation Babych and Hartley (2003), relation extraction Yu et al. (2019), entity linking Xue et al. (2019), and so on.

Recently, NER has seen remarkable advances with the help of pre-trained representation models, such as BERT Devlin et al. (2019) and XLNet Yang et al. (2019). Providing contextual representation, these pre-trained models could be easily applied to NER applications as an encoder by just fine-tuning it. Despite refreshing the state-of-the-art performance of NER, the current pre-training techniques are not directly optimized for NER. Typically, these models build unsupervised training objectives to capture dependency between words and learn a general language representation Tian et al. (2020), while rarely considering incorporating named entity information which can provide rich knowledge for NER. Due to little knowledge connection between NER and general language modeling, how to adapt public pre-trained models to be NER-specific remains an open problem.

To this end, injecting named entity knowledge during pre-training is a possible solution. However, this process of knowledge acquisition may be inefficient and expensive. In fact, there are extensive weakly labeled annotations that naturally exist on the web yet to be explored for NER model pre-training, which are relatively easier to obtain compared with labeled data Cao et al. (2019). One can collect them from online resources, such as the Wikipedia anchors and gazetteers (named entity dictionaries). Although automatically derived corpora usually contain massive noisy data, it still contains some extend the valuable semantic information required for NER Peng et al. (2019).

In this paper, we propose a _Coarse-to-Fine Entity knowledge Enhanced_ (CoFEE) pre-training framework for NER task, aiming to gather and utilize knowledge related to named entities. In particular, we first extract anchors from Wikipedia and use them as training corpora for entity span identification. While anchors have no entity type information, the model could get general-typed entity

knowledge from them and learn to distinguish entity words and non-entity words. In the second phase, we use gazetteers and anchors to generate weakly labeled data for specific entity types and use it to train the model for extracting entities with coarse-grained type. Furthermore, another observation is that entities with the same coarse-grained type may belong to different fine-grained types. According to the cluster hypothesis [15], the features of entities with the same latent fine-grained label will cluster together in the semantic space. Intuitively, mining these latent cluster structures provides auxiliary information about the coarse-grained entity type, which could be beneficial to improve the NER performance. Based on such motivation, we finally devise a self-supervised method to exploit fine-grained type knowledge and tap the potential of weakly labeled data, which effectively train the NER model with clustering-generated pseudo labels.

We conduct experiments on three realistic NER benchmarks in this paper. Experimental results show that the proposed CoFEE pre-training framework significantly outperforms other competitive baselines, often by large margins. We also demonstrate that CoFEE pre-training can work well in more challenging, label-free and low-resource scenarios. Further ablation studies show the impact of each pre-training task in achieving these strong performance. To the best of our knowledge, this is the first work that has tackled NER-specific representation during pre-training.

## 2 Related Work

Entity Knowledge for NER.Recently, neural networks have been used for NER and achieved great success [30, 46, 87, 9]. Specifically, various types of entity knowledge, including lexical words, gazetteers and anchors in Wikipedia have been proved to be useful for a wide range of sentiment analysis tasks.

For supervised NER task, some researchers utilize lattice structure to incorporate the lexical information into character-based NER and avoid the segmentation error propagation of word [14, 15, 16, 17, 18, 19]. Additionally, gazetteers have long been regarded as a piece useful knowledge for NER, previous methods commonly incorporated gazetteers by either using them as handcraft features [13, 16] or auxiliary structural information [15, 16].

For weakly supervised NER, a typical line of methods centres around transfer learning to extract source knowledge for target, such as crossdomain [14, 15, 16] or cross-lingual [13, 15, 16, 17]. There are also a lot of weak labels lying on the web or gazetteers, which have not been explored. Consequently, a number of works focus on distantly supervised methods, using anchors or gazetteers to generate data by distant supervision [16, 17, 18, 19, 20].

Task Specific Pre-training.Unsupervised language model pre-training and task-specific fine-tuning achieve SOTA results on many NLP tasks, including NER [21, 15, 16]. Recently, with the help of automatically minded knowledge lying in the web, researchers devoted them to the pre-training models for specific tasks, including word sense disambiguation [8], word-in-context tasks [14], entity-linking and relation classification [15], sentiment classification [15].

## 3 Background

In this section, we give a brief introduction to MRC-NER [13], which achieves satisfying performance in NER and thus is chosen as the foundation of our work. Given an input paragraph \(X=\{x_{1},x_{2},\cdots,x_{n}\}\) where \(x_{i}\) denotes the i-th character, NER aims at discovering each entity \(x_{start,end}\) in \(X\) and identify its corresponding type \(y\in Y\), where \(Y\) is the set of predefined tags(e.g., PER, LOC). \(x_{start,end}=\{x_{start},x_{start+1},\cdots,x_{end-1},x_{end}\}\) is a substring of \(X\) satisfying _start \(\leq\) end_. Specifically, MRCNER formulates NER as a machine reading comprehension (MRC) problem. Each entity type \(y\) is characterized by a natural language query \(Q_{y}=\{q^{y}_{1},q^{y}_{2},...,q^{y}_{m}\}\), and entities are extracted by answering these queries given the contexts. For example, the task of assigning the PER label to "[Washington] was born into slavery on the farm" is formalized as answering the question "Find person including fictional". This strategy naturally introduces the natural language query which encodes significant prior knowledge about the enti 

category to extract.

Formally, MRC-NER model concatenates the query \(Q\) and paragraph \(X\), forming string {[CLS], \(Q,[\text{SEP}],X\)}, where [CLS] and [SEP] are special tokens. Then BERT Devlin et al. (2019) captures the contextual information for each token in the string via self-attention and produces the representation matrix \(\mathbf{H}\in\mathbb{R}^{n\times d}\) of \(X\), where \(d\) is the dimension of the last layer of BERT. To extract entity spans, the representation of each word is fed to two softmax layers to predict the probability of each token being a start or end index as follows:

\[P_{\text{start}}(\mathbf{y}_{i}^{\mathrm{s}}|x_{i})=softmax( \mathbf{W}_{\mathrm{s}}\mathbf{h}_{i}+\mathbf{b}_{\mathrm{s}}), \tag{1}\] \[P_{\text{end}}(\mathbf{y}_{i}^{\mathrm{e}}|\mathbf{x}_{i})= softmax(\mathbf{W}_{\mathrm{e}}\mathbf{h}_{i}+\mathbf{b}_{\mathrm{e}}), \tag{2}\]

where \(\mathbf{W}_{\mathrm{s}},\mathbf{W}_{\mathrm{e}}\in\mathbb{R}^{d\times 2}\) and \(\mathbf{b}_{\mathrm{s}},\mathbf{b}_{\mathrm{e}}\in\mathbb{R}^{2}\) are trainable parameters. At training time, \(S\) associated with each question \(Q_{y}\) is paired with two label sequences \(\mathbf{Y}_{\text{start}}=\{y_{1}^{s},y_{2}^{s},...,y_{n}^{s}\}\) and \(\mathbf{Y}_{\text{end}}=\{y_{1}^{s},y_{2}^{s},...,y_{n}^{s}\}\), where \(y_{i}^{s}\) (\(y_{i}^{s}\)) is the ground-truth label of \(x_{i}\) being the start (end) index of a \(y\)-typed entity or not. The cross-entropy loss of start and end index predictions are therefore denoted as:

\[\mathcal{L}_{\text{start}}^{\mathcal{D}}=-\frac{1}{n}\sum_{i=1}^{ n}y_{i}^{s}log(P_{\text{start}}(y_{i}^{s}|x_{i})), \tag{3}\] \[\mathcal{L}_{\text{end}}^{\mathcal{D}}=-\frac{1}{n}\sum_{i=1}^{ n}y_{i}^{s}log(P_{\text{end}}(y_{i}^{s}|x_{i})), \tag{4}\]

where \(\mathcal{D}\) denotes the training dataset. Finally, the overall training objective to be minimized can be formulated as follows:

\[\mathcal{L}_{\text{MRC}}^{\mathcal{D}}=\mathcal{L}_{\text{start}}^{\mathcal{ D}}+\mathcal{L}_{\text{end}}^{\mathcal{D}}. \tag{5}\]

## 4 Methodology

In this section, we introduce the overall framework of our coarse-to-fine pre-training. Figure 1 gives a brief illustration, which operates in three stages as follows: (1) Stage 1: identity entity span based on Wikipedia anchors; (2) Stage 2: extract coarse-grained entities based on gazetteers; (3) Stage 3: predict fine-grained entity types with a clustering-oriented self-supervised method.

### Entity Span Identification

Pre-trained language Models such as BERT Devlin et al. (2019) and XLNet Yang et al. (2019) have been proven to capture rich language information from text. However, as the entity information of a text is seldom explicitly studied, it is hard to expect such pre-trained general representations to capture entity-centric knowledge. In order to better capture entity information and learn NER-specific representation, we propose the first pre-training task named Entity Span Identification (ESI). The entity-centric knowledge is automatically mined from the large scale Wikipedia corpus. In Wikipedia, an anchor \(\langle m,e\rangle\) links a mention \(m\) to an entity \(e\). Therefore, we assign an "Entity" tag to each anchor in the sentence and construct a General-typed weakly labeled NER dataset \(\mathcal{D}_{\text{g}}\) without considering the entity type. To align with MRC-NER, the question of the generated dataset is set as "_Find Entities_". With the general labeled data, the MRC-NER model can be warmed-up with loss \(\mathcal{L}_{\text{MRC}}^{\mathcal{D}_{\text{f}}}\). By integrating the general-typed named entity knowledge into the pre-training process, the learned representation would be incorporated with the structural information of crucial importance for NER.

### Named Entity Extraction

After the ESI pre-training, the model has learned to distinguish entity words and non-entity words. Then we step into the second phase (i.e., NEE) in which the model is trained to extract typed entities with gazetteer-labeled data. To alleviate human

Figure 1: The overall architecture of CoFEE. In Fine-grained Entity Typing, the _solid line_ represents the training phase and the _dotted line_ represents the clustering phase. These two stages are iteratively done until the network converges.

 

effort, gazetteer-based distant supervision has been applied to automatically generate labeled data and has gained successes in NER Yang et al. (2018); Peng et al. (2019). A standard strategy is to scan through the anchor text in \({\mathcal{D}_{\text{g}}}\) using the gazetteer of a given entity type \(y\) and treat anchors matched with entries of the given gazetteer as the entities with type \(y\). In this way, we can obtain a specific-typed NER dataset \({\mathcal{D}_{s}}\), which is then exploited to train the MRC-NER model by optimizing \({\mathcal{L}^{\mathcal{D}_{s}}_{\text{MRC}}}\). Besides, in order to meet the paradigm of MRC-NER, we also generate a natural language query for each entity type. This procedure is critical since queries encode prior knowledge about labels. Inspired by Li et al. (2020), we take annotation guideline notes as references to construct queries and illustrate all of the queries used in our model in Table 1. They are theoretical description of the tag categories, thus having the ability to make the model incorporate the information within the label categories unambiguously and completely.

However, as most existing gazetteers only cover part of entities, the automatically derived dataset usually contains massive noisy data including missing labels, incorrect boundaries and types. To address this issue, we propose an iterative self-picking strategy. At the beginning (iteration 0), the model starts with training from the original noisy label set. At the end of each iteration, the model determines the next label set by making predictions on \({\mathcal{D}_{\text{s}}}\). Concretely, a new entity will be extracted with type \(y\) if the probabilities of its start and end indices being predicted as \(y\) are both greater than a picking threshold \(\delta\). In the next iteration, we use the new derived dataset as input for the model training. Considering that we aim to recall the missing labels, we set \(\delta<0.5\). The model is trained until we find the best model w.r.t. the performance on the validation set. And the final derived dataset is denoted as \({\mathcal{D}_{\text{s}}^{\text{best}}}\).

### Fine-grained Entity Typing

NEE pre-training focuses on teaching the model named entity knowledge about coarse-grained entity types. However, one coarse-grained entity type may be composed of a set of fine-grained entity types. For example, the coarse-grained type _Location_ includes City, Country, Bodies of water, etc. These fine-grained types can provide auxiliary information to help us understand the meaning of _Location_. With this in mind, it is intuitive to group the extracted entities with a cluster miner, and use the subsequent cluster assignments as pseudo labels to mine the fine-grained NER knowledge. One of the most well-studied clustering algorithms is k-Means, and the simplicity and efficiency have established it as a popular means for performing clustering across different disciplines.

Formally, in order to partition the entity set \(E=\{e_{1},e_{2},\cdots,e_{M}\}\) in \({\mathcal{D}_{\text{s}}^{\text{best}}}\) into pre-defined \(K\) distinct clusters \(\{C_{k}\}_{k=1}^{K}\), k-Means minimizes the sum of the intra-cluster variances \(\sum_{k=1}^{K}{\mathcal{V}_{k}}\), where \({\mathcal{V}_{k}}=\sum_{i=1}^{M}\delta_{ik}||{\mathbf{e}_{i}}-{\mathbf{m}_{k} }||^{2}\) and \({\mathbf{m}_{k}}=\sum_{i=1}^{M}\delta_{ik}{\mathbf{e}_{i}}/\sum_{i=1}^{M} \delta_{ik}\) are the variance and the center of the k-th cluster, respectively, \({\mathbf{e}_{i}}=sumpool([{\mathbf{h}_{start_{i}}},{\mathbf{h}_{start_{i}+1}} ,...,{\mathbf{h}_{end_{i}}}])\) denotes the representation of the \(i\)-th entity, and \(\delta_{ik}\) is a cluster indicator variable with \(\delta_{ik}=1\) if \(e_{i}\in C_{k}\)

\begin{table}
\begin{tabular}{|l|l|} \hline Type & Question \\ \hline PER & \({\mathcal{J}_{\text{g}}}{\mathbb{H}_{\text{g}}}{\mathbb{H}_{\text{g}}}{ \mathbb{H}_{\text{g}}}{\mathbb{H}_{\text{g}}}{\mathbb{H}_{\text{g}}}{ \mathbb{H}_{\text{g}}}{\mathbb{H}_{\text{g}}}{\mathbb{H}_{\text{g}}}{ \mathbb{H}_{\text{g}}}{\mathbb{H}_{\text{g}}}{\mathbb{H}_{\text{g}}}{ \mathbb{H}_{\text{g}}}{\mathbb{H}_{\text{g}}}{\mathbb{H}_{\text{g}}}{\mathbb{ H}_{\text{g}}}{\mathbb{H}_{\text{g}}}{\mathbb{H}_{\text{g}}}{\mathbb{H}_{\text{g}}}{ \mathbb{H}_{\text{g}}}{\mathbb{H}_{\text{g}}}{\mathbb{H}_{\text{g}}}{\mathbb{H}_{ \text{g}}}{\mathbb{H}_{\text{g}

 

and \(0\) otherwise. Clustering proceeds by alternating between assigning instances to their closest center and recomputing the centers, until a local minimum is reached. The cluster assignments are used as pseudo labels to guide the transformation of \({\mathcal{D}_{s}^{\text{best}}}\) to a pseudo-labeled fine-grained dataset \({\mathcal{D}_{\text{c}}=\{(e_{i},y_{i}^{\text{c}})\}}\), where \(y_{i}^{\text{c}}\) is the pseudo label of \(e_{i}\) Then we can take the negative log-likelihood of the pseudo-labeled tags as the training objective:

\[P_{c}(y_{i}^{\text{c}}|e_{i}) =softmax(\mathbf{W_{\text{c}}}\mathbf{e_{i}+b_{\text{c}}}) \tag{6}\] \[{\mathcal{L}_{\text{clus}}} =-\frac{1}{M}\sum_{i=1}^{M}y_{i}^{\text{c}}log(P_{\text{c}}(y_{ i}^{\text{c}}|e_{i})) \tag{7}\]

where \(\mathbf{W_{\text{c}}}\in\mathbb{R}^{K\times d}\) and \(\mathbf{b_{\text{c}}}\in\mathbb{R}^{K}\) are trainable parameters, \(P_{\text{c}}(y_{i}^{\text{c}}|e_{i})\) denotes the probability of entity \(e_{i}\) being predicted to the \(y_{i}^{\text{c}}\)-th cluster. Recall that our purpose is to pre-train the NER model to discover typed entities belonging to \(Y\) rather than fine-grained entities, so \({\mathcal{L}_{\text{clus}}}\) can be deemed as an auxiliary task to assist the model to mine the fine-grained NER knowledge and regularize the optimization of \({\mathcal{L}_{\text{MRC}}^{\mathcal{D}_{\text{s}}}}\). So the training objective in this stage is defined as:

\[{\mathcal{L}_{\text{FET}}}={\mathcal{L}_{\text{MRC}}^{\mathcal{ D}_{\text{s}}^{\text{best}}}}+\gamma{\mathcal{L}_{\text{clus}}}, \tag{8}\]

where \(\gamma\) is the trade-off parameter.

While optimizing with pseudo labels created by the cluster miner seems reasonable, the inevitable label noise caused by the clustering procedure is ignored. To this end, we propose a variance-weighted cross-entropy loss to alleviate the influence of noisy pseudo labels. Obviously, the inverse of \(\mathcal{V}_{k}\) (\(\mathcal{V}_{k}^{-1}\)) represents the intra-cluster compactness of the \(k\)-th cluster. If the features of instances in the \(k\)-th cluster are close together, \(\mathcal{V}_{k}^{-1}\) will be large, meantime the confidence of assigning pseudo label \(y_{i}\) to these instances should also be high and vice versa. Thus we re-formulate Equation 7 as:

\[{\mathcal{L}_{\text{clus}}} =-\frac{1}{M}\sum_{i=1}^{M}\alpha_{y_{i}^{\text{c}}}y_{i}^{ \text{c}}log(P_{\text{c}}(y_{i}^{\text{c}}|e_{i})), \tag{9}\] \[\alpha_{y_{i}^{\text{c}}} =\frac{exp(\mathcal{V}_{y_{i}^{\text{c}}}^{-1})}{\sum_{k=1}^{K} exp(\mathcal{V}_{n}^{-1})}. \tag{10}\]

Finally, we iterate the above clustering-optimizing process by putting back the model to output new representations, generate new pseudo labels \({\mathcal{D}_{\text{c}}}\) and start the next iteration.

```
0: Wikipedia corpus; Specific typed gazetteers;
0: Specific typed validation data \({\mathcal{D}_{i}^{val}}\);
0: Initialize Model Parameters \(\theta\) with BERT.
1: Construct \({\mathcal{D}_{\text{g}}}\) based on Wikipedia anchors
2:for\(epoch\gets 1\) to \(e_{1}\)do\(\triangleright\) Stage 1.
3: Update \(\theta\) w.r.t. \({\mathcal{L}_{\text{MRC}}^{\mathcal{D}_{\text{g}}}}\)
4:endfor
5:Construct \({\mathcal{D}_{\text{v}}}\) by matching \({\mathcal{D}_{\text{g}}}\) to gazetteet.
6:for\(epoch\leftarrow 1\) to \(e_{2}\)do\(\triangleright\) Stage 2.
7: Update \(\theta\) w.r.t. \({\mathcal{L}_{\text{MRC}}^{\mathcal{D}_{\text{s}}}}\).
8:if\(score(\theta,{\mathcal{D}_{\text{v}}^{val}})>best\_score\)then
9:\(\theta_{\text{test}}\leftarrow\theta\); \({\mathcal{D}_{\text{v}}^{val}}\leftarrow{\mathcal{D}_{\text{v}}}\);
10:\(best\_score=score(\theta,{\mathcal{D}_{\text{u}}^{val}})\)
11:endif
12: Re-label \({\mathcal{D}_{\text{v}}}\) with \(\theta\).
13:endfor
14:\(\theta\leftarrow\theta_{\text{best}}\), \(best\_score\gets 0\)
15: Construct \({\mathcal{D}_{\text{v}}}\) by clustering entities in \({\mathcal{D}_{\text{v}}^{\text{best}}}\).
16:for\(epoch\leftarrow 1\) to \(e_{3}\)do\(\triangleright\) Stage 3.
17: Update \(\theta\) w.r.t. \({\mathcal{L}_{\text{FET}}}\)
18:if\(score(\theta,{\mathcal{D}_{\text{v}}^{val}})>best\_score\)then
19:\(\theta_{\text{best}}\leftarrow\theta\);
20:\(best\_score=score(\theta,{\mathcal{D}_{\text{s}}^{val}})\)
21:endif
22: Re-cluster entities in \({\mathcal{D}_{\text{s}}^{\text{best}}}\) and construct new \({\mathcal{D}_{\text{c}}}\)
23:endfor
24:return\(\theta_{\text{best}}\)
```

**Algorithm 1** Coarse-to-fine Pre-training

### Algorithm Workflow

In this subsection, we introduce the overall procedure of our framework. Algorithm 1 gives the scratch. First, we construct general-typed NER data \({\mathcal{D}_{\text{g}}}\) based on Wikipedia anchors, and pre-train the model to extract general typed entities with loss \({\mathcal{L}_{\text{MRC}}^{\mathcal{D}_{\text{s}}}}\). Then we leverage the gazetteer-based distant supervision strategy to construct a specific-typed NER dataset \({\mathcal{D}_{\text{s}}}\), and propose an iterative self-picking method to alleviate the data missing problem. In each iteration, the model is optimized to fit the data labeled by the previous iteration. When the performance on the validation set starts to decline, the iteration is ended and the best-performed model is passed to the third stage, where a cluster miner is deployed to group the entities extracted from the second stage into fine-grained types, and the model is trained to simultaneously distinguish fine-grained entities and extract specific-typed entities. Also, we iteratively cluster the features from the last iteration to gradually refine the fine-grained pseudo labels for current.

## 5 Experiments

We evaluate the CoFEE framework under two settings: (i) supervised setting (ii) weakly supervised setting. In the supervised setting, the pre-trained 

model is fine-tuned on human-labeled datasets while in the weakly-supervised setting, the model pre-trained with CoFEE is directly applied to perform NER without fine-tuning. Next, we describe these experiments in detail.

### Datasets

Our experiments are conducted on three benchmarks. (1) **Chinese Ontonotes 4.0** consists of newswire text and published by Ralph et al. (2011). It is annotated by four types: PER (Person), ORG (Organization), GPE (Geo-Political Entity) and LOC (Location) for Chinese named entity. It contains 15.7k sentences for training and 4.3k for testing. (2) **E-commerce** is a Chinese NER dataset collected from the e-commerce domain and released by Ding et al. (2019). It is annotated by PROD (product) and BRAN (brand) types. The training and test datasets contain 273k and 53k lines, respectively. (3) **Twitter** is an English NER dataset (Qi et al., 2018), following Peng et al. (2019), we only use textual information to perform NER and make entity detection on PER, LOC and ORG. It contains 4,000 tweets for training and 3,257 tweets for testing.

### Pre-training Corpora

Wikipedia.We use 20200401 Chinese and English Wikipedia dumps23 for data construction, where we set the max sentence length as 250 and remove the sentences which contain three or fewer anchors. The resulting Chinese corpora contains 1,116,514 sentences and 6,383,142 anchors (entity mentions), and the English corpora contains 3,911,059 sentences and 37,755,176 anchors.

Footnote 2: [https://dumps.wikimedia.org/zhwiki/20200401/zhwiki-20200401-pages-articles.xml.bz2](https://dumps.wikimedia.org/zhwiki/20200401/zhwiki-20200401-pages-articles.xml.bz2)

Footnote 3: [https://dumps.wikimedia.org/enwiki/20200401/enwiki-20200401-pages-articles.xml.bz2](https://dumps.wikimedia.org/enwiki/20200401/enwiki-20200401-pages-articles.xml.bz2)

Gazetteer.For Chinese PER, ORG, GPE, and LOC, we collect the gazetteers from the crowdsource dictionaries used by Chinese Input Method "Sougou"4, which contain 2,314 person names, 2,649 organization names, 895 geopolitical entities, and 628 location names. For Chinese PROD and BRAN, we use the gazetteers provided by Ding et al.(2019), which contain 628 brand names and 2,974 product names. For English PER, ORG and LOC, we collect the gazetteers using the method released by Peng et al.(2019), which contain 2,795 person names, 1,825 organization names and 1,408 location names.

Footnote 4: [https://pinyin.soogu.com/dict/](https://pinyin.soogu.com/dict/)

### Baselines

We chose two types of baselines: supervised methods and the weakly supervised methods. We call our proposed CoFEE pre-training framework with MRC-NER backbone as CoFEE-MRC. In addition, to demonstrate the model-agnostic and generic property of CoFEE, we also implemented another competitive baseline by replacing the MRC-NER backbone with a widely used BERT model Devlin et al. (2019) without any change in the training procedure, denoted as CoFEE-BERT. We used open-source release of [https://github.com/huggingface/transformers](https://github.com/huggingface/transformers).

Supervised Setting.We fine-tune CoFEE-MRC and CoFEE-BERT on supervised NER data and compare with the following baselines to learn how improvement can be achieved for supervised models. **BiLSTM-CRF**Huang et al. (2015) is a classical neural-network-based baseline for NER, which usually achieves competitive performance in supervised NER. **BERT-Tagger**Devlin et al. (2019) uses the outputs from the last layer of model BERt\({}_{base}\) as the character-level enriched contextual representations to make sequence labeling. **MRC-NER**Li et al. (2020) formulates NER as a machine reading comprehension task and uses BERT as the basic encoder.

Weakly Supervised Setting.We investigate the effect of CoFEE-MRC for solving the NER task without any human annotations, and compare the model to some weakly supervised NER models. For fair comparison, we implemented baselines with the same gazetteers constructed in Section 5.2. **gaetteer Matching** applies the constructed gazetteers to the test set directly to obtain entity mentions with exactly the same surface name. By comparing with it, we can check the improvements of neural models over the distant supervision itself. **MRC-NER** uses the MRC-NER backbone to perform weakly supervised NER task with gazetteer labeled training data. Furthermore, we explore the influence of our proposed pre-training tasks by removing entity span identification pre-training (**-ESI**) and fine-grained entity typing pre-training (**-FTP**) from CoFEE-MRC 



### Hyper-parameter settings

We use the BertAdam as our optimizer, all of the models are implemented under PyTorch using a single NVIDIA Tesla V100 GPU, we use "bertbase-chinese" and "bert-base-cased" as our pretrained models for Chinese and English language, the number of parameters is same to these pretrained models in addition to two binary classifier. For each training stage, we vary the learning rate from \(1e-6\) to \(1e-4\). In NEE stage, we select the best trade-off from \(0.1\) to \(0.5\) with an incremental \(0.1\). In FET stage, we choose the number of clusters \(K\) from \(\{K-2,K-1,K,K+1,K+2\}\) if we set \(K\) as the categories of fine-grained entity. For all these hyper-parameters, we select the best according to the F1-score on the dev sets.

### Evaluation

Following the evaluation metrics in previous work Li et al. (2020), we apply the entity-level (exact entity match) standard micro Precision (P), Recall (R), and F1 score to evaluate the results.

### Overall Performance

Table 2 contains results for models tuned on human-labeled NER data. We can observe that our CoFEE-MRC pre-training performs remarkably better than MRC-NER, establishing an impressive new state-of-the-art for supervised NER on OntoNotes and Twitter of \(82.64\%\) and \(73.86\%\), respectively. CoFEE-BERT also significantly improves the performance compared with BERT and achieves a new SOTA for supervised NER on E-commerce of \(79.73\%\), which confirms the model-agnostic property of our CoFEE pre-training framework. Please note that the results of MRC-NER on OntoNotes have a few concerns need to be addressed. MRC-NER set the max sentence length as 77, which is far less than the true maximum length of the dataset. While in our method, we promise that the maximum length is more than 100.

Table 3 reports the results of our models against to baselines under the weakly supervised setting. We can find that: **1)** Gazetteer Matching performs quite poorly and the capability of this method is strongly influenced by the size of the gazetteers. For OntoNotes, the coverage of the large scale gazetteer is almost \(40\%\), but also its huge size causes the low precision. For Twitter, the recall value is about \(14\%\) due to its limited gazetteers. **2)** If we directly use MRC-NER to perform weakly supervised NER task with gazetteer labeled data, the model achieves a degree of improvement but is still inaccurate due to the distantly labeled data. **3)** CoFEE-MRC achieves the state-of-the-art F1 score on all three benchmarks, which confirms the validity of our proposed CoFEE pre-training framework. **4)** FET pre-training task brings performance improvements, whi

\begin{table}
\begin{tabular}{|l c c c|} \hline \multicolumn{4}{|c|}{Chinese OntoNotes 4.0} \\ \hline Model & P & R & F1 \\ \hline BiLSTM-CRF Huang et al. (2015) & 72.0 & 75.1 & 73.5 \\ BERT Devlin et al. (2019) & 78.01 & 80.35 & 79.16 \\ MRC-NER Li et al. (2020) & **82.98** & 81.25 & 82.11 \\ \hline CoFEE-BERT & 80.27 & 80.64 & 80.46 \\ CoFEE-MRC & 82.5 & **82.78** & **82.64** \\ \hline \hline \multicolumn{4}{|c|}{**E-commerce**} \\ \hline Model & P & R & F1 \\ \hline BiLSTM-CRF Huang et al. (2015) & 71.1 & 76.1 & 73.6 \\ BERT Devlin et al. (2019) & 77.06 & **80.65** & 78.81 \\ MRC-NER Li et al. (2020) & 79.47 & 78.3 & 78.88 \\ \hline CoFEE-BERT & 79.13 & 80.34 & **79.73** \\ CoFEE-MRC & **80.26** & 78.88 & 79.56 \\ \hline \hline \multicolumn{4}{|c|}{**Twitter**} \\ \hline Model & P & R & F1 \\ \hline BiLSTM-CRF Huang et al. (2015) & –

ness of the introducing fine-grained named entity knowledge. **5)** ESI pre-training further improves the performance, which demonstrates the necessity to warm-up the pre-trained language model using general-typed named entity knowledge.

## 6 Analysis

### Impact of Data Size

We analyze the influence of reducing the amount of pre-training data and fine-tuning data. The results on the dev set of E-commerce are shown in Figure 2(a) and 2(b), respectively. From Figure 2(a), we can observe that increasing the size of the pre-training data will improve the performance generally, but the improvement tends to flatten out with \(60\%\sim 80\%\) data. We suppose that this is because of the number of unique patterns, the influence of the training data size has its local minimum and maximum critical point. From Figure 2(b), we see that knowledge enhanced pre-training is more effective for low-resource cases, where there is a larger gap in performance between our CoFEEMRC and MRC-NER. Besides, the performance of CoFEE pre-training is more stable as data scale changes. This further demonstrates that our CoFEE pre-training framework can significantly reduce human efforts to create NER taggers.

### Impact of Picking Rate

We then evaluate the influence of the value and variation of our picking rate \(\delta\). From Figure 2(c), we can see that setting a lower picking rate to recall more named entities can indeed improve a great performance for the model and gives the highest result with \(\delta_{0}=0.1\).

### Impact of Gazetteer Size

We further explore the change of the training data and performance when we use gazetteers of different sizes. In particular, we used 20%, 40%, 60%, 80% and 100% of the original gazetteers to construct pre-training corpora. Statistical information of each resultant gazetteer is illustrated in Figure 3(a), and the model performance on the E-commerce dev set with these gazetteers is demonstrated in Figure 3(b). We can observe that increasing the size of gazetteers will generally improve the performance of our proposed CoFEE-MRC model and the performance growths in line with the performance of "Matching", indicating that in addition to the gazetteer size, matching degree also has a crucial influence on the model performance.

### Impact of Cluster Size

The proposed CoFEE framework does require a cluster size \(K\) as the scope for pseudo labels. One may wonder whether the choice of \(K\) has a significant influence on the final results. In this subsection, we vary \(K\) from 4 to 90 and report the F1 score of CoFEE-MRC on the E-commerce dev set. As shown in Figure 2(d), the best performance is obtained when \(K\) is exactly set as the number of fine-grained entity types described in the queries (23), indicating that our CoFEE pre

Figure 3: Statistic information and model performance with gazetteers of different sizes on the Weibo dev set.

Figure 2: (a) Impact of pre-training data size on the weakly supervised setting; and (b) Impact of fine-tuning data size on the supervised setting; and (c) Impact of picking rate \(\delta\); and (d) Impact of cluster size \(K\).

 

training can leverage this information as useful prior knowledge. Thanks to the self-supervised learning schema, when we very from 3 to 90, the model achieves stable F1 score and is not sensitive to the choice of \(K\). The results also further indicate the applicability of the proposed framework when being applied to a new kind of named entity where the number of fine-grained entity types is not available in advance. We can safely assign a larger value than needed and the model is still robust.

## 7 Conclusion

We investigated coarse-to-fine entity knowledge enhanced pre-training for named entity recognition, which integrates three kinds of entity knowledge with different granularity levels. Though conceptually simple, our framework is highly effective and easy to implement. On three popular NER benchmarks, we found consistent improvements over both state-of-the-art supervised and weakly-supervised methods. Further analysis verifies the necessity of utilizing NER knowledge for pre-training models.

## Acknowledgements

We would like to thank the anonymous reviewers for their insightful comments and suggestions. This research is supported by the National Key Research and Development Program of China (grant No.2016YFB0801003) and the Strategic Priority Research Program of Chinese Academy of Sciences (grant No.XDC02040400).

## References

* Alan et al. (2011) Ritter Alan, Clark Sam, Etzioni Oren, and et al. 2011. Named Entity Recognition in Tweets: An Experimental Study. In ACL, pp. 1524-C1534. Cited by: SS1.
* B. Babych and S. Hartley (2003)Improving machine translation quality with automatic named entity recognition. In EMMT workshop, pp. 1-8. Cited by: SS1.
* Y. Cao, Z. Hu, T. Chua, Z. Liu, and H. Ji (2019)Low-resource name tagging learned with weakly labeled data. In EMNLP-IJCNLP, pp. 261-270. Cited by: SS1.
* O. Chapelle, B. Scholkopf, and A. Zien (2009)Semi-supervised learning (chapelle, o. et al., eds.; 2006)[book reviews]. IEEE Transactions on Neural Networks20 (3), pp. 542-542. External Links: Document Cited by: SS1.
* R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa (2011)Natural language processing almost from scratch. Journal of Machine Learning Research, pp. 2493-2537. Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: Pre-training of deep bidirectional transformers for Language Understanding.. In NAACL, pp. 4171-4186. Cited by: SS1.
* R. Ding, P. Xie, X. Zhang, W. Lu, L. Li, and L. Si (2019)A neural multi-digraph model for chinese ner with gazetteers. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1462-1467. Cited by: SS1.
* S. Dominic, D. Tatiana, D. Corro Luciano, H. Johannes, and W. Gerhard (2018)A study of the importance of external knowledge in the named entity recognition task. In ACL, pp. 241-246. Cited by: SS1.
* D. Freitag (2004)Trained named entity recognition using distributional clusters. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pp. 262-269. Cited by: SS1.
* T. Gui, R. Ma, Q. Zhang, L. Zhao, Y. Jiang, and X. Huang (2019)CNN-based chinese ner with lexicon rethinking. In IJCAI, pp. 4982-4988. Cited by: SS1.
* T. Gui, Y. Zou, Q. Zhang, M. Peng, J. Fu, Z. Wei, and X. Huang (2019)A Lexicon-Based graph neural network for chinese ner. In EMNLP, pp. 1039-1049. Cited by: SS1.
* L. Huang, C. Sun, X. Qiu, and X. Huang (2019)GlossBERT: BERT for word sense disambiguation with gloss knowledge. In EMNLP-IJCNLP, pp. 3509-3514. Cited by: SS1.
* Z. Huang, W. Liang Xu, and K. Yu (2015)Bidirectional LSTM-CRF models for sequence tagging. arXiv: Computation and Language. Cited by: SS1.
* C. Jia, X. Liang, and Y. Zhang (2019)Cross-domain ner using cross-domain language modeling. In ACL, pp. 2464-2474. Cited by: SS1.
* Y. Levine, B. Lenz, Or. Dagan, D. Padnos, O. Sharir, S. Shalevshwartz, A. Shashua, and Y. Shoham (2020)SenseBERT: driving some sense into bert. In ACL, Cited by: SS1.
* X. Li, J. Feng, Y. Meng, Q. Han, F. Wu, and J. Li (2020)A unified mrc framework for named entity recognition. ACL. Cited by: SS1.
* B. Yuchen Lin and W. Lu (2018)Neural adaptation layers for cross-domain named entity recognition. In EMNLP, pp. 2012-2022. Cited by: SS1.
* S. Liu, B. Tang, Q. Chen, and X. Wang (2015)Effects of semantic features on machine learning-based drug name recognition systems: Word embeddings vs. Manually constructed micionaries. Information6 (4), pp. 848-865.

 

Tianyu Liu, Jin-Ge Yao, and Chin-Yew Lin. 2019. Towards Improving Neural Named Entity Recognition with Gazetteers. In _ACL_, pages 5301-5307.
* Ma and Hovy (2016) Xuezhe Ma and uard Hovy. 2016. End-to-end Sequence Labeling via Bi-directional LSTM-CNs-CRF. In _ACL_, pages 1064-1074.
* Ni et al. (2017) Jian Ni, Georgiana Dinu, and Radu Florian. 2017. Weakly Supervised Cross-Lingual Named Entity Recognition via Effective Annotation and Representation Projection. In _ACL_, pages 1470-1480.
* Peng et al. (2019) Minlong Peng, Xiaoyu Xing, Qi Zhang, Jinlan Fu, and Xuanjing Huang. 2019. Distantly Supervised Named Entity Recognition using Positive-Unlabeled Learning. In _ACL_, pages 2409-2419.
* Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. In _NAACL_, pages 2227-2237.
* Qi et al. (2018) Zhang Qi, Fu Jinlan, Liu Xiaoyu, and Huang Xuanjing. 2018. Adaptive Co-attention Network for Named Entity Recognition in Tweets. In _AAAI_, pages 5674-5681.
* Ralph et al. (2011) Weischedel Ralph, Pradhan Sameer, Ramshaw Lance, Palmer Martha, Xue Nianwen, Marcus Mitchell, Taylor Ann, Greenberg Craig, Hovy Eduard, Belvin Robert, and et al. 2011. Ontonotes release 4.0. In _LDC2011TO3, Philadelphia, Penn.: Linguistic Data Consortium_.
* dos Santos and Guimaraes (2015) Cicero dos Santos and Victor Guimaraes. 2015. Boosting Named Entity Recognition with Neural Character Embeddings. In _Proceedings of the Fifth Named Entity Workshop_, pages 25-33.
* Sui et al. (2019) Dianbo Sui, Yubo Chen, Kang Liu, Jun Zhao, and Shengping Liu. 2019. Leverage Lexical Knowledge for Chinese Named Entity Recognition via Collaborative Graph Network. In _EMNLP_, pages 3821-3831.
* Tian et al. (2020) Hao Tian, Can Gao, Xinyan Xiao, Hao Liu, Bolei He, Hua Wu, Haifeng Wang, and Feng Wu. 2020. SKEP: Sentiment Knowledge Enhanced Pretraining for Sentiment Analysis. _arXiv preprint arXiv:2005.05635_.
* Xie et al. (2018) Jiateng Xie, Zhilin Yang, Graham Neubig, and Jaime Smith, Noah A. andCarbonell. 2018. Neural Crossingual Named Entity Recognition with Minimal Resources. In _EMNLP_, pages 369-379.
* Xue et al. (2019a) Mengge Xue, Weiming Cai, Jinsong Su, Linfeng Song, Yubin Ge, Yukao Liu, and Bin Wang. 2019a. Neural Collective Entity Linking Based on Recurrent Random Walk Network Learning. In _IJCAI_, pages 5327-5333.
* Xue et al. (2019b) Mengge Xue, Bowen Yu, Tingwen Liu, Erli Meng, and Bin Wang. 2019b. Porous Lattice Transformer Encoder for Chiense NER. _arXiv: Computation and Language_.
* Yang et al. (2018) Yaosheng Yang, Wenliang Chen, Zhenghua Li, Zhengqiu He, and Min Zhang. 2018. Distantly Supervised NER with Partial Annotation Learning and Reinforcement Learning. In _COLING_, pages 2159-2169.
* Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized Autoregressive Pretraining for Language Understanding. In _Advances in neural information processing systems_, pages 5754-5764.
* Yang et al. (2017) Zhilin Yang, Ruslan Salakhutdinov, and William W Cohen. 2017. Transfer Learning for Sequence Tagging with Hierarchical Recurrent Networks. In _ICLR_.
* Yu et al. (2019) Bowen Yu, Zhenyu Zhang, Tingwen Liu, Bin Wang, Sujian Li, and Quangang Li. 2019. Beyond Word Attention: Using Segment Attention in Neural Relation Extraction. In _IJCAI_, pages 5401-5407.
* Zhang and Yang (2018) Yue Zhang and Jie Yang. 2018. Chinese NER Using Lattice LSTM. In _ACL_, pages 1554-1564.
* Zhang et al. (2019) Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced Language Representation with Informative Entities. In _ACL_, pages 1441-1451.
* Zhou et al. (2019) Joey Tianyi Zhou, Hao Zhang, Di Jin, Hongyuan Zhu, Meng Fang, Rick Siow Mong Goh, and Kenneth Kwok. 2019. Dual Adversarial Neural Transfer for Low-Resource Named Entity Recognition. In _ACL_, pages 3461-3471.