SlowGuess/NAACL_2022_OCR huggingface dataset:
Yingjie Gu<sup>1</sup>, Xiaoye Qu<sup>1*</sup>, Zhefeng Wang<sup>1†</sup>, Yi Zheng<sup>1</sup>, Baoxing Huai<sup>1</sup>, Nicholas Jing Yuan<sup>1</sup>
<sup>1</sup>Huawei Cloud, China
{guyingjie4,quxiaoye,wangzhefeng,zhengyi29}@huawei.com huaibaoxing@huawei.com,nicholas.jing.yuan@gmail.com
Recent years have witnessed the improving performance of Chinese Named Entity Recognition (NER) from proposing new frameworks or incorporating word lexicons. However, the inner composition of entity mentions in character-level Chinese NER has been rarely studied. Actually, most mentions of regular types have strong name regularity. For example, entities end with indicator words such as "公司 (company)" or "银行 (bank)" usually belong to organization. In this paper, we propose a simple but effective method for investigating the regularity of entity spans in Chinese NER, dubbed as Regularity-Inspired reCOgnition Network (RICON). Specifically, the proposed model consists of two branches: a regularity-aware module and a regularity-agnostic module. The regularity-aware module captures the internal regularity of each span for better entity type prediction, while the regularity-agnostic module is employed to locate the boundary of entities and relieve the excessive attention to span regularity. An orthogonality space is further constructed to encourage two modules to extract different aspects of regularity features. To verify the effectiveness of our method, we conduct extensive experiments on three benchmark datasets and a practical medical dataset. The experimental results show that our RICON significantly outperforms previous state-of-the-art methods, including various lexicon-based methods.
Named entity recognition (NER) aims at identifying text spans pertaining to specific entity types. It plays an important role in many downstream tasks such as relation extraction (Cheng et al., 2021), entity linking (Gu et al., 2021), co-reference resolution (Clark and Manning, 2016), and knowledge graph (Ji et al., 2020). Due to the complex composition (Gui et al., 2019), character-level Chinese NER
(The Niger River flows through Niger into the Gulf of Guinea)
(a) [尼日尔河]流经尼日尔与尼日尼亚 Overlap
Regularity: (XX + 河River)
(Chinese players did very well in this competition)
(b) [中国]队员们在本次比赛中取得了优异成绩
Regularity: (XX + 队Team)
Figure 1: (a) Complex composition of Chinese NER and regularity. (b) Excessive focusing on regularity leads to wrong entity boundary.
is more challenging compared to English NER. As shown in Figure 1 (a), the middle character "流" can constitute words with the characters to both their left and their right, such as "河流 (River)" and "流经 (flows)", leading to ambiguous character boundaries.
There are two typical frameworks for NER. The first one conceptualizes NER as a sequence labeling task (Huang et al., 2015; Lample et al., 2016; Ma and Hovy, 2016), where each character is assigned to a special label (e.g., B-LOC, I-LOC). The second one is span-based method (Li et al., 2020a; Yu et al., 2020), which classifies candidate spans based on their span-level representations. However, despite the success of these two types of methods, they do not explicitly take the complex composition of Chinese NER into consideration. Recently, several works (Zhang and Yang, 2018; Gui et al., 2019; Li et al., 2020b) utilize external lexicon knowledge to help connect related characters and promote capturing the local composition. Nevertheless, building the lexicon is time-consuming and the quality of the lexicon may not be satisfied.
In contrast to previous works, we observe that the regularity exists in the common NER types (e.g., ORG and LOC). As shown in Figure 1 (a), "尼日尔河(Niger River)"follows the specific composition pattern "XX+河  $(\mathrm{XX} + \mathrm{River})$  "which ends
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1863 - 1873
with indicator character "河" and mostly belongs to location type, and the ambiguous character "流" can properly constitute "流经" with the right character "经". Thus, the regularity information serves as important clues for entity type recognition and identifying the character composition. Formally, we refer to regularity as specific internal patterns contained in a type of entity (Lin et al., 2020). However, too immersed regularity leads to unfavorable boundary detection of entities and disturbing character composition. As shown in Figure 1 (b), "中国队 (Chinese team)" conforms to the pattern "XX+队 (XX + Team)", but the correct entity boundary should be "中国 (Chinese)" and "队员 (players)" according to the context. Therefore, the context also plays a key role in determining the character boundary.
In this paper, we introduce a simple but effective method to explore the regularity information of entity spans for Chinese NER, dubbed as Regularity-Inspired reCOgnition Network (RICON). The proposed model consists of two branches named regularity-aware module and regularity-agnostic module, where each module has task-specific encoder and optimization object. Concretely, the regularity-aware module aims at analyzing the internal regularity of each span and integrates the significant regularity information into the corresponding span-level representation, leading to precise entity type prediction. Meanwhile, the regularity-agnostic module is devised to capture context information and avoid excessive focus on intra-span regularity. Furthermore, we adopt an orthogonality space restriction to encourage two branches to extract different features with regard to the regularity. To verify the effectiveness of our method, we conduct extensive experiments on three large-scale benchmark datasets (OntoNotes V4.0, OntoNotes V5.0, and MSRA). The results show that RICON achieves considerable improvements compared to the state-of-the-art models, even outperforming existing lexicon-based models. Moreover, we experiment on a practical medical dataset (CBLUE) to further demonstrate the ability of RICON.
- This is the first work that explicitly explores the internal regularity of entity mentions for Chinese NER.
- We propose a simple but effective method for Chinese NER, which effectively utilizes reg-
uality information while avoiding excessive focus on intra-span regularity.
- Extensive experiments on three large-scale benchmark datasets and a practical medical dataset demonstrate the effectiveness of our proposed method.
Traditional methods treat NER as a sequence labeling task, where each word or character in the sentence is assigned to a special label. As a representative, Huang et al. (2015) utilized the BiLSTM as an encoder to learn the contextual representation, and then exploited Conditional Random Field (CRF) as a decoder to label the tokens. The BiLSTM-CRF architecture achieved superior performance on various datasets, hence many following works (Lample et al., 2016; Ma and Hovy, 2016) adopt such architecture. More recently, strong pre-trained language models such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019) are incorporated to further enhance the performance of NER. Although the sequence labeling framework achieves decent performance on flat NER, it struggles for nested NER. As a result, span-based models are proposed to solve the nested problem by classifying all possible spans into predefined types (e.g. PER, LOC) in the sentence. For example, Yu et al. (2020) adopted a biaffine attention model to assign scores for all potential spans and achieved the state-of-the-art performance on both flat and nested English NER datasets. Shen et al. (2021) also employed span-based framework on Chinese NER datasets. In this paper, we adopt span-based method as our basic framework for two reasons. Firstly, the span-based method considers each span and naturally suits analyzing inner-span character composition. Secondly, the span-based framework can easily extend our method from flat NER to nested NER.
Recently, for Chinese NER, researchers proposed various lexicon-based models that incorporate the external lexicon information and obtained better results. Zhang and Yang (2018) investigated Lattice-LSTM for incorporating word lexicons into the character-based NER model. However, the lattice structure fails to compute in parallel. To address this problem, Gui et al. (2019) introduced a lexicon-based graph neural network that recasts Chinese NER as a node classification task. There are also several works that focus on incorporating all matched words from the lexicon into the charac
Figure 2: Overall structure of RICON. Each character in the sentence is first embedded by BERT. Then, two separate Bi-LSTM layers are adopted to encode representations for the regularity-aware module and regularity-agnostic module. An orthogonality space is further utilized to encourage extracting different features for each module.
ter embeddings (Ma et al., 2020; Liu et al., 2021). Different from the aforementioned lexicon-based works that incorporate external resources, in this paper, we focus on exploring the internal regularity information of spans.
The overall architecture of our RICON is shown in Figure 2, which mainly consists of two branches: the regularity-aware module and the regularity-agnostic module.
3.1 Embedding and Task-specific Encoder
First of all, each character of the input sequence is embedded into a dense vector. Then the character vectors are separately fed into two task-specific bidirectional LSTM (BiLSTM) layers to extract the corresponding hidden states for each module respectively. Formally, given a sentence with  $l$  characters  $s = \{c_1, c_2, \dots, c_l\}$ . We use a standard BERT (Devlin et al., 2019) to obtain the context dependent embeddings for a target token:
$$
x _ {i} = \operatorname {B E R T} \left(c _ {i}\right) \tag {1}
$$
Then, the sequence of character embeddings will be fed to two separate BiLSTM layers for regularity-aware module and regularity-agnostic module. The hidden state of BiLSTM is expressed as follows:
$$
\overrightarrow {h} _ {i, \tau} = \overrightarrow {\mathrm {L S T M}} \left(x _ {i}, \overrightarrow {h} _ {i - 1, \tau}\right) \tag {2}
$$
Figure 3: Conceptual comparison of three architectures for span-based NER.  $\{, \}$  denotes the representations across from  $i$ th to  $j$ th character of the span  $s_{i,j}$ .
$$
\overleftarrow {h} _ {i, \tau} = \overleftarrow {\mathrm {L S T M}} \left(x _ {i}, \overleftarrow {h} _ {i - 1, \tau}\right) \tag {3}
$$
$$
h _ {i, \tau} = \left[ \overrightarrow {h} _ {i, \tau}; \overleftarrow {h} _ {i, \tau} \right] \tag {4}
$$
where  $\tau \in \{\text{aware, agnostic}\}$ , [;] denotes concatenation, and the dimension of  $h_{i,\tau}$  is  $2d$ . The character sequence representation can be denoted as  $H_{\tau} = \{h_{1,\tau}, \dots, h_{i,\tau}, \dots, h_{l,\tau}\}$ .
3.2 Regularity-aware Module
In this module, we aim to explore the internal regularity of each span. As shown in Figure 3 (a), typical span-based NER methods (Sohrab and Miwa, 2018; Xia et al., 2019; Li et al., 2020a) represent each entity span via concatenating corresponding head and tail features, and use a linear classifier
to predict the type of this span. In this way, the span features are coarse-grained. Then, as denoted in Figure 3 (b), Yu et al. (2020) propose a biaffine decoder to enhance the interaction between head and tail representations after two MLPs and predict span types simultaneously. Nevertheless, the internal regularity among characters in the span is still neglected in this biaffine method.
Consequently for this, our regularity-aware module is devised to capture the internal regularity feature for each span  $s_{i,j}$ , as demonstrated in Figure 3 (c). It is worth noting that span representations are obtained by the head and tail characters of the span, while the regularity representations stem from each character in the span. To achieve this goal, we utilize a linear attention to obtain the regularity representation of each span as follows:
$$
a _ {t} = W _ {\mathrm {r e g}} ^ {\top} h _ {t} + b _ {\mathrm {r e g}} \tag {5}
$$
$$
\alpha_ {t} = \frac {\exp \left(a _ {t}\right)}{\sum_ {k = i} ^ {j} \exp \left(a _ {k}\right)} \tag {6}
$$
$$
h _ {s _ {i, j}} ^ {(\mathrm {r e g})} = \sum_ {t = i} ^ {j} \alpha_ {t} \cdot h _ {t} \tag {7}
$$
where  $h_t = h_{t,\mathrm{aware}}$  and  $t \in \{i, i + 1, \dots, j\}$  is the index of the span,  $W_{\mathrm{reg}} \in \mathbb{R}^{2d \times 1}$  and  $b_{\mathrm{reg}} \in \mathbb{R}^1$  are learnable weights and bias respectively. For a span whose length is 1, we do not extract extra features but use the hidden representation  $h_{i,\mathrm{aware}}$  to denote its regularity. The regularity feature  $H^{(\mathrm{reg})} \in \mathbb{R}^{l \times l \times 2d}$  will be used for the subsequent entity type prediction.
To predict the type of an entity, our model integrates the regularity feature of each span into the span representations. Firstly, we acquire the span representation via a biaffine attention mechanism by interacting head and tail features:
$$
h _ {s _ {i, j}} ^ {(\text {s p a n})} = h _ {i} ^ {\top} U ^ {(1)} h _ {j} + \left(h _ {i} \oplus h _ {j}\right) U ^ {(2)} + b _ {1} \tag {8}
$$
where  $h_i, h_j \in H_{\mathrm{aware}}$  are the head and tail representations of span  $s_{i,j}$ .  $U^{(1)}$  is a  $2d \times 2d \times 2d$  tensor,  $U^{(2)}$  is a  $4d \times 2d$  matrix, and  $b_1$  is the bias. It is worth noting that here we do not apply two separate MLPs like Figure 3 (b) to generate different representations for the head and tail features of the spans, as different MLPs will project the head, tail, and regularity representation into distinct spaces. The experiment also verifies that such space inconsistency degrades the recognition performance.
Then a gated network is devised to integrate the span and regularity representation as below:
$$
g _ {s _ {i, j}} = \sigma \left(U ^ {(3)} \left[ h _ {s _ {i, j}} ^ {(\text {s p a n})}; h _ {s _ {i, j}} ^ {(\text {r e g})} \right] + b _ {2}\right) \tag {9}
$$
$$
h _ {s _ {i, j}} = g _ {s _ {i, j}} \odot h _ {s _ {i, j}} ^ {(\text {s p a n})} + (1 - g _ {s _ {i, j}}) \odot h _ {s _ {i, j}} ^ {(\text {r e g})} \tag {10}
$$
where  $U^{(3)}\in \mathbb{R}^{4d\times 1}$  is a trainable parameter and  $b_{2}$  is the bias.  $\sigma$  denotes the sigmoid function and  $\odot$  mean the element-wise dot multiplication. Finally, we adopt a standard linear classifier with a softmax function to predict the entity type for each span.
$$
\tilde {y} _ {s _ {i, j}} = \operatorname {S o f t m a x} \left(W _ {\text {t y p e}} ^ {\top} h _ {s _ {i, j}} + b _ {3}\right) \tag {11}
$$
where  $W_{\mathrm{type}} \in \mathbb{R}^{2d \times c}$  is a trainable parameter and  $b_{3}$  is the bias. The loss function of the regularity-aware module is defined as cross-entropy:
$$
\mathcal {L} _ {\text {a w a r e}} = - \frac {1}{N} \sum_ {n = 1} ^ {N} \sum_ {i = 1} ^ {l} \sum_ {j = 1} ^ {l} y _ {s _ {i, j}} ^ {(n)} \log \left(\tilde {y} _ {s _ {i, j}} ^ {(n)}\right), i \leq j \tag {12}
$$
where  $\tilde{y}_{s_{i,j}}$  denotes the prediction and  $y_{s_{i,j}}$  is the ground truth type of the span.  $N$  is the number of training samples in the regularity-aware module.
3.3 Regularity-agnostic Module
By considering regularity, above regularity-aware module makes the model stricter in terms of predicting the entity type, thus improving the precision of entity prediction. Nevertheless, too immersed regularity may result in inaccurate word boundaries. To get rid of it, we propose to erase the concrete form of golden entities and relieve the excessive learning of structural pattern by regularity-aware module. In this scenario, the head and tail features which determine boundary become more significant, thereby we first apply two multi-layer perceptrons (MLPs) on the hidden states from BiLSTM to get separate representations for head and tail. Then a biaffine decoder is leveraged for obtaining entity probability of the span  $s_{i,j}$  as follows:
$$
\bar {h} _ {i} = \mathrm {M L P} _ {\text {h e a d}} (h _ {i}) \quad \bar {h} _ {j} = \mathrm {M L P} _ {\text {t a i l}} (h _ {j}) \tag {13}
$$
$$
\bar {y} _ {i j} = \sigma \left(\left[ \bar {h} _ {i}; 1 \right] ^ {\top} U _ {m} \left[ \bar {h} _ {j}; 1 \right]\right) \tag {14}
$$
where  $h_i = h_{i,\mathrm{agnostic}}$ ,  $h_j = h_{j,\mathrm{agnostic}}$ ,  $U_m$  is a  $(2d + 1) \times 1 \times (2d + 1)$  trainable parameter,  $\sigma$
is the sigmoid function. Finally, we adopt binary cross-entropy loss to train this task.
$$
\begin{array}{l} \mathcal {L} _ {\text {a g n o s t i c}} = - \frac {1}{N} \sum_ {n = 1} ^ {N} \sum_ {i = 1} ^ {l} \sum_ {j = 1} ^ {l} \left[ y _ {i j} ^ {(n)} \log \left(\bar {y} _ {i j} ^ {(n)}\right) \right. \tag {15} \\ + (1 - y _ {i j} ^ {(n)}) \log (1 - \bar {y} _ {i j} ^ {(n)}), i \leq j \\ \end{array}
$$
where  $\bar{y}_{ij}$  denotes the prediction and  $y_{ij}$  is the binary target indicating whether the span is an entity or not.  $N$  is the number of training samples in the regularity-agnostic module.
3.4 Orthogonality Space Restriction
As regularity-aware module aims to capture the regularity information while regularity-agnostic module pays no attention to the concrete regularity, we expect to learn different features for these two modules. To this end, we construct an orthogonality space on the top of two BiLSTM layers to encourage encoding different aspects of the input embeddings. The loss is calculated as follows:
$$
H _ {\text {o r t h}} = H _ {\text {a w a r e}} ^ {\top} H _ {\text {a g n o s t i c}} \tag {16}
$$
$$
\mathcal {L} _ {\text {o r t h}} = \left\| H _ {\text {o r t h}} \right\| _ {F} ^ {2} = - \frac {1}{N} \sum_ {n = 1} ^ {N} \sum_ {i = 1} ^ {l} \sum_ {j = 1} ^ {l} \left| h _ {i j} ^ {(n)} \right| ^ {2} \tag {17}
$$
where  $\| \cdot \| _F^2$  is the squared Frobenius norm and  $N$  is the number of training elements.
During training, our RICON can be trained by joint optimizing above three sub-tasks, so we define the total loss as below:
$$
\mathcal {L} = \lambda_ {1} \mathcal {L} _ {\text {a w a r e}} + \lambda_ {2} \mathcal {L} _ {\text {a g n o s t i c}} + \lambda_ {3} \mathcal {L} _ {\text {o r t h}} \tag {18}
$$
where  $\lambda_{1},\lambda_{2}$  , and  $\lambda_{3}$  are hyperparameter. During inference, we directly use regularity-aware module to predict the entity type for each span and apply a post-processing constraint for two overlapped entity candidates  $E_{1}$  and  $E_{2}$  that if  $E_{1i} <   E_{2i}\leq$ $E_{1j} <   E_{2j}$  , where  $i$  and  $j$  are start and end indexes, we only select the entity with the higher type score.
OntoNotes V4.0 (Weischedel et al., 2011). It is a multilingual corpus in the news domain. This
dataset has 4 entity types. We use the same split as (Zhang and Yang, 2018).
OntoNotes V5.0 (Pradhan et al., 2013). Compared with V4.0, this version has more news data and contains 18 types of entities. We use the same split as (Jie and Lu, 2019).
MSRA (Levow, 2006). It contains 3 types of named entities collected from the news domain. We use the same split as (Gui et al., 2019).
CBLUE-CMeEE (Hongying et al., 2020). CBLUE is Chinese biomedical language understanding evaluation which consists of 10 sub-tasks. Among them, CMeEE focuses on Chinese medical entity extraction and has 9 types of entities. We use the official train and dev split.
In addition, all types of OntoNotes V4.0, OntoNotes V5.0, MSRA, and 8 types of CBLUE-CMeEE are flat NER, while the symptom type of CBLUE-CMeEE is nested NER.
Due to the space limitation, the statistics of all datasets are listed in the appendix.
In our experiments, we use the same settings for all datasets. Specifically, we adopt the standard pre-trained Chinese BERT-base model with 768 dimensions hidden representation to obtain character embeddings. We use Adam optimizer with 2e-5 learning rate for BERT embedding fine-tuning and 0.001 learning rate for other parts. The number of layer and dropout rate of BiLSTM encoders are set to 3 and 0.4. The hidden state size of BiLSTM encoders is set to 200. For the regularity-agnostic module, the output dimension of MLPs and the dropout rate are set to 150 and 0.2. To avoid overfitting, we also apply 0.1 dropout rate for the BERT output embeddings. For the hyper-parameters in loss, we set  $\lambda_1 = \lambda_2 = 1$  and  $\lambda_3 = 0.5$ . For all experiments including ablation study, we adopt an average of performance over five different runs to reduce randomness.
4.3 Comparison Methods
In our experiments, we compare our RICON with recent state-of-the-art methods, where part of them contain pre-trained language model BERT or external Chinese lexicon information. Here we briefly describe five typical methods:
(1) Star-GAT (Chen and Kong, 2021) propose a Star-transformer based NER system. They utilize explicit head and tail boundary information and
Models	Lexicon	OntoNotes V4.0	OntoNotes V5.0	MSRA
P	R	F1	P	R	F1	P	R	F1
Lattice LSTM (Zhang and Yang, 2018)	✓	76.35	71.56	73.88	-	-	-	93.57	92.79	93.18
Collaborative Graph Network (Sui et al., 2019)	✓	75.06	74.52	74.79	-	-	-	94.01	92.93	93.47
LGN (Gui et al., 2019)	✓	76.13	73.68	74.89	-	-	-	94.19	92.73	93.46
DGLSTM-CRF (Jie and Lu, 2019)	-	-	-	77.40	77.41	77.40	-	-	-
WC-GCN (Tang et al., 2020)	✓	76.59	75.17	75.87	-	-	-	94.82	93.98	94.40
Star-GAT (Chen and Kong, 2021)	79.25	80.66	79.95	78.22	80.88	79.53	-	-	-
with Pre-trained Language Model
BERT-Tagger	76.01	79.96	77.93	73.59	80.55	76.91	93.40	94.12	93.76
BERT+LSTM+CRF	81.99	81.65	81.82	77.12	79.81	78.44	95.06	94.61	94.83
BERT+PLTE (Mengge et al., 2020)	✓	79.62	81.82	80.60	-	-	-	94.91	94.15	94.53
BERT+Biaffine (Yu et al., 2020)	81.06	84.03	82.52	78.79	80.07	79.43	96.65	94.75	95.20
BERT+FLAT (Li et al., 2020b)	✓	-	-	81.82	-	-	-	-	-	96.09
BERT+SoftLexicon (Ma et al., 2020)	✓	83.41	82.21	82.81	-	-	-	95.75	95.10	95.42
LEBERT (Liu et al., 2021)	✓	-	-	82.08	-	-	-	-	-	95.70
RICON (Ours)	81.95	84.78	83.33	79.26	81.64	80.43	95.94	96.33	96.14
Dependency GAT-based implicit boundary information to improve the performance. It is the SOTA model on the OntoNotes V5.0 dataset.
(2) BERT+Biaffine (Yu et al., 2020) recast NER as a task of identifying start and end positions and assigning a type to each span by a biaffine attention.
(3) BERT+FLAT (Li et al., 2020b) devise a FLAT model for Chinese NER, which converts the lattice structure into a flat structure consisting of spans to overcome the shortage of lattice-based model (Zhang and Yang, 2018). They also equipped with BERT embeddings and achieved the SOTA performance on the MSRA dataset.
(4) BERT+SoftLexicon (Ma et al., 2020) incorporate the word lexicon into the character features. They leverage Chinese lexicon to match every character in the sentence with word appeared in the lexicon to improve the performance, which achieves the SOTA performance on OntoNotes V4.0.
(5) LEBERT (Liu et al., 2021) introduce a Lexicon Adapter layer to integrate external lexicon knowledge into BERT layers directly.
We present the results on three benchmark datasets in Table 1. From this table, we can observe that our RICON achieves the state-of-the-art performance on these datasets. Moreover, RICON even outperforms recent methods with Chinese lexicon significantly. Concretely, on OntoNotes V4.0, RICON achieves 0.81 absolute F1 improvement over the strong method BERT+Biaffine and 0.52 absolute improvement compared with the SOTA lexicon-based method BERT+SoftLexicon. On OntoNotes V5.0, we obtain a decent improvement compared to the SOTA approach Star-GAT by 0.90 F1 score. In addition, on MSRA, although the
Table 1: We compare our RICON with recent state-of-the-art models on three Chinese benchmark datasets.
All Types	Symptom Type
Model	P	R	F1	P	R	F1
BERT-Tagger	53.41	63.32	57.95	40.57	45.38	42.84
BERT-CRF	58.34	64.08	61.07	46.01	47.51	46.75
BERT-Biaffine	64.17	61.29	62.29	63.17	33.91	44.14
RICON	66.25	64.89	65.57	57.93	43.99	50.01
Table 2: Performance of models on CBLUE-CMeEE, including all types and symptom type.
improvement of our model over the SOTA model BERT-FLAT is limited, our model still surpasses the other two lexicon-based models LEBERT and BERT+SoftLexicon by 0.44 and 0.72 respectively.