SlowGuess/ACL_2020_OCR huggingface dataset:
Ruotian Ma $^{1*}$ , Minlong Peng $^{1*}$ , Qi Zhang $^{1,3}$ , Zhongyu Wei $^{2,3}$ , Xuanjing Huang $^{1}$
$^{1}$ Shanghai Key Laboratory of Intelligent Information Processing,
$^{2}$ School of Data Science, Fudan University
$^{3}$ Research Institute of Intelligent and Complex Systems, Fudan University
{rtma19,mlpeng16,qz,zywei,xjhuang} @fudan.edu.cn
Recently, many works have tried to augment the performance of Chinese named entity recognition (NER) using word lexicons. As a representative, Lattice-LSTM (Zhang and Yang, 2018) has achieved new benchmark results on several public Chinese NER datasets. However, Lattice-LSTM has a complex model architecture. This limits its application in many industrial areas where real-time NER responses are needed.
In this work, we propose a simple but effective method for incorporating the word lexicon into the character representations. This method avoids designing a complicated sequence modeling architecture, and for any neural NER model, it requires only subtle adjustment of the character representation layer to introduce the lexicon information. Experimental studies on four benchmark Chinese NER datasets show that our method achieves an inference speed up to 6.15 times faster than those of state-of-the-art methods, along with a better performance. The experimental results also show that the proposed method can be easily incorporated with pre-trained models like BERT.
Named Entity Recognition (NER) is concerned with the identification of named entities, such as persons, locations, and organizations, in unstructured text. NER plays an important role in many downstream tasks, including knowledge base construction (Riedel et al., 2013), information retrieval (Chen et al., 2015), and question answering (Diefenbach et al., 2018). In languages where words are naturally separated (e.g., English), NER has been conventionally formulated as a sequence
labeling problem, and the state-of-the-art results have been achieved using neural-network-based models (Huang et al., 2015; Chiu and Nichols, 2016; Liu et al., 2018).
Compared with NER in English, Chinese NER is more difficult since sentences in Chinese are not naturally segmented. Thus, a common practice for Chinese NER is to first perform word segmentation using an existing CWS system and then apply a word-level sequence labeling model to the segmented sentence (Yang et al., 2016; He and Sun, 2017b). However, it is inevitable that the CWS system will incorrectly segment query sentences. This will result in errors in the detection of entity boundary and the prediction of entity category in NER. Therefore, some approaches resort to performing Chinese NER directly at the character level, which has been empirically proven to be effective (He and Wang, 2008; Liu et al., 2010; Li et al., 2014; Liu et al., 2019; Sui et al., 2019; Gui et al., 2019b; Ding et al., 2019).
A drawback of the purely character-based NER method is that the word information is not fully exploited. With this consideration, Zhang and Yang, (2018) proposed Lattice-LSTM for incorporating word lexicons into the character-based NER model. Moreover, rather than heuristically choosing a word for the character when it matches multiple words in the lexicon, the authors proposed to preserve all words that match the character, leaving the subsequent NER model to determine which word to apply. To realize this idea, they introduced an elaborate modification to the sequence modeling layer of the LSTM-CRF model (Huang et al., 2015). Experimental studies on four Chinese NER datasets have verified the effectiveness of Lattice-LSTM.
However, the model architecture of Lattice-LSTM is quite complicated. In order to introduce lexicon information, Lattice-LSTM adds several additional edges between nonadjacent characters
*Equal contribution.  
<sup>1</sup>The source code of this paper is publicly available at https://github.com/v-mipeng/LexiconAugmentedNER.
5951
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5951-5960 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
in the input sequence, which significantly slows its training and inference speeds. In addition, it is difficult to transfer the structure of LatticeLSTM to other neural-network architectures (e.g., convolutional neural networks and transformers) that may be more suitable for some specific tasks.
In this work, we propose a simpler method to realize the idea of Lattice-LSTM, i.e., incorporating all the matched words for each character to a character-based NER model. The first principle of our model design is to achieve a fast inference speed. To this end, we propose to encode lexicon information in the character representations, and we design the encoding scheme to preserve as much of the lexicon matching results as possible. Compared with Lattice-LSTM, our method avoids the need for a complicated model architecture, is easier to implement, and can be quickly adapted to any appropriate neural NER model by adjusting the character representation layer. In addition, ablation studies show the superiority of our method in incorporating more complete and distinct lexicon information, as well as introducing a more effective word-weighting strategy. The contributions of this work can be summarized as follows:
- We propose a simple but effective method for incorporating word lexicons into the character representations for Chinese NER.
- The proposed method is transferable to different sequence-labeling architectures and can be easily incorporated with pre-trained models like BERT (Devlin et al., 2018).
We performed experiments on four public Chinese NER datasets. The experimental results show that when implementing the sequence modeling layer with a single-layer Bi-LSTM, our method achieves considerable improvements over the state-of-the-art methods in both inference speed and sequence labeling performance.
In this section, we introduce several previous works that influenced our work, including the Softword technique and Lattice-LSTM.
2.1 Softword Feature
The Softword technique was originally used for incorporating word segmentation information into downstream tasks (Zhao and Kit, 2008; Peng and
Dredze, 2016). It augments the character representation with the embedding of its corresponding segmentation label:
$$
x _ {j} ^ {c} \leftarrow \left[ x _ {j} ^ {c}; e ^ {\operatorname {s e g}} \left(\operatorname {s e g} \left(c _ {j}\right)\right) \right]. \tag {1}
$$
Here,  $seg(c_j) \in \mathcal{Y}_{seg}$  denotes the segmentation label of the character  $c_{j}$  predicted by the word segmentor,  $e^{seg}$  denotes the segmentation label embedding lookup table, and typically  $\mathcal{V}_{seg} = \{\mathrm{B},\mathrm{M},\mathrm{E},\mathrm{S}\}$ .
However, gold segmentation is not provided in most datasets, and segmentation results obtained by a segmenter can be incorrect. Therefore, segmentation errors will inevitably be introduced through this approach.
2.2 Lattice-LSTM
Lattice-LSTM designs to incorporate lexicon information into the character-based neural NER model. To achieve this purpose, lexicon matching is first performed on the input sentence. If the subsequence  $\{c_i,\dots ,c_j\}$  of the sentence matches a word in the lexicon for  $i < j$ , a directed edge is added from  $c_{i}$  to  $c_{j}$ . All lexicon matching results related to a character are preserved by allowing the character to be connected with multiple other characters. Intrinsically, this practice converts the input form of a sentence from a chain into a graph.
In a normal LSTM layer, the hidden state  $h_i$  and the memory cell  $c_i$  of each time step is updated by:
$$
h _ {i}, c _ {i} = f \left(h _ {j - 1}, c _ {j - 1}, x _ {j} ^ {c}\right), \tag {2}
$$
However, in order to model the graph-based input, Lattice-LSTM introduces an elaborate modification to the normal LSTM. Specifically, let  $s_{<*,j>}$  denote the list of sub-sequences of sentence  $s$  that match the lexicon and end with  $c_j$ ,  $h_{<*,j>}$  denote the corresponding hidden state list  $\{h_i, \forall s_{<i,j} \in s_{<*,j}\}$ , and  $c_{<*,j>}$  denote the corresponding memory cell list  $\{c_i, \forall s_{<i,j} \in s_{<*,j}\}$ . In Lattice-LSTM, the hidden state  $h_j$  and memory cell  $c_j$  of  $c_j$  are now updated as follows:
$$
h _ {j}, c _ {j} = f \left(h _ {j - 1}, c _ {j - 1}, x _ {j} ^ {c}, s _ {<   *, j >}, h _ {<   *, j >}, c _ {<   *, j >}\right), \tag {3}
$$
where  $f$  is a simplified representation of the function used by Lattice-LSTM to perform memory update.
From our perspective, there are two main advantages to Lattice-LSTM. First, it preserves all the possible lexicon matching results that are related to
5952
a character, which helps avoid the error propagation problem introduced by heuristically choosing a single matching result for each character. Second, it introduces pre-trained word embeddings to the system, which greatly enhances its performance.
However, efficiency problems exist in Lattice-LSTM. Compared with normal LSTM, Lattice-LSTM needs to additionally model  $s_{<*,j >}, h_{<*,j >}$  and  $c_{<*,j >}$  for memory update, which slows the training and inference speeds. Additionally, due to the complicated implementation of  $f$ , it is difficult for Lattice-LSTM to process multiple sentences in parallel (in the published implementation of Lattice-LSTM, the batch size was set to 1). These problems limit its application in some industrial areas where real-time NER responses are needed.
In this work, we sought to retain the merits of Lattice-LSTM while overcoming its drawbacks. To this end, we propose a novel method in which lexicon information is introduced by simply adjusting the character representation layer of an NER model. We refer to this method as SoftLexicon. As shown in Figure 1, the overall architecture of the proposed method is as follows. First, each character of the input sequence is mapped into a dense vector. Next, the SoftLexicon feature is constructed and added to the representation of each character. Then, these augmented character representations are put into the sequence modeling layer and the CRF layer to obtain the final predictions.
3.1 Character Representation Layer
For a character-based Chinese NER model, the input sentence is seen as a character sequence  $s = \{c_1, c_2, \dots, c_n\} \in \mathcal{V}_c$ , where  $\mathcal{V}_c$  is the character vocabulary. Each character  $c_i$  is represented using a dense vector (embedding):
$$
x _ {i} ^ {c} = e ^ {c} \left(c _ {i}\right), \tag {4}
$$
where  $e^c$  denotes the character embedding lookup table.
Char + bichar. In addition, Zhang and Yang, (2018) has proved that character bigrams are useful for representing characters, especially for those methods not using word information. Therefore, it is common to augment the character representations with bigram embeddings:
$$
x _ {i} ^ {c} = \left[ e ^ {c} \left(c _ {i}\right); e ^ {b} \left(c _ {i}, c _ {i + 1}\right) \right], \tag {5}
$$
Figure 1: The overall architecture of the proposed method.
where  $e^b$  denotes the bigram embedding lookup table.
3.2 Incorporating Lexicon Information
The problem with the purely character-based NER model is that it fails to exploit word information. To address this issue, we proposed two methods, as described below, to introduce the word information into the character representations. In the following, for any input sequence  $s = \{c_1, c_2, \dots, c_n\}$ ,  $w_{i,j}$  denotes its sub-sequence  $\{c_i, c_{i+1}, \dots, c_j\}$ .
ExSoftword Feature
The first conducted method is an intuitive extension of the Softword method, called ExSoftword. Instead of choosing one segmentation result for each character, it proposes to retain all possible segmentation results obtained using the lexicon:
$$
\boldsymbol {x} _ {j} ^ {c} \leftarrow \left[ \boldsymbol {x} _ {j} ^ {c}; e ^ {\text {s e g}} (\operatorname {s e g s} (c _ {j}) \right], \tag {6}
$$
where  $\text{segs}(c_j)$  denotes all segmentation labels related to  $c_j$ , and  $e^{\text{seg}}(\text{segs}(c_j))$  is a 5-dimensional multi-hot vector with each dimension corresponding to an item of  $\{B, M, E, S, O\}$ .
As an example presented in Figure 2, the character  $c_7$  ("西") occurs in two words,  $w_{5,8}$  ("中山西路") and  $w_{6,7}$  ("山西"), that match the lexicon, and it occurs in the middle of "中山西路" and the end of "山西". Therefore, its corresponding segmentation result is  $\{\mathbf{M},\mathbf{E}\}$ , and its character representation is enriched as follows:
$$
\boldsymbol {x} _ {7} ^ {c} \leftarrow \left[ \boldsymbol {x} _ {7} ^ {c}; e ^ {\operatorname {s e g}} (\{M, E \}) \right]. \tag {7}
$$
5953
Figure 2: The ExSoftword method.
Here, the second and third dimensions of  $e^{seg}(\cdot)$  are set to 1, and the rest dimensions are set to 0.
The problem of this approach is that it cannot fully inherit the two merits of Lattice-LSTM. First, it fails to introduce pre-trained word embeddings. Second, it still loses information of the matching results. As shown in Figure 2, the constructed ExSoftword feature for characters  $\{c_5, c_6, c_7, c_8\}$  is  $\{\{\mathrm{B}\}, \{\mathrm{B}, \mathrm{M}, \mathrm{E}\}, \{\mathrm{M}, \mathrm{E}\}, \{\mathrm{E}\}\}$ . However, given this constructed sequence, there exists more than one corresponding matching results, such as  $\{w_{5,6} (\text{"中山")}, w_{5,7} (\text{"中山西")}, w_{6,8} (\text{"山西路")}\}$  and  $\{w_{5,6} (\text{"中山")}, w_{6,7} (\text{"山西")}, w_{5,8} (\text{"中山西路")}\}$ . Therefore, we cannot tell which is the correct result to be restored.
SoftLexicon
Based on the analysis on Exsoftword, we further developed the SoftLexicon method to incorporate the lexicon information. The SoftLexicon features are constructed in three steps.
Categorizing the matched words. First, to retain the segmentation information, all matched words of each character  $c_{i}$  is categorized into four word sets "BMES", which is marked by the four segmentation labels. For each character  $c_{i}$  in the input sequence  $= \{c_{1}, c_{2}, \dots, c_{n}\}$ , the four set is constructed by:
$$
\begin{array}{l} \mathrm {B} (\mathrm {c _ {i}}) = \{\mathrm {w _ {i , k}}, \forall \mathrm {w _ {i , k}} \in \mathrm {L}, \mathrm {i} <   \mathrm {k} \leq \mathrm {n} \}, \\ \mathrm {M} \left(\mathrm {c} _ {\mathrm {i}}\right) = \left\{\mathrm {w} _ {\mathrm {j}, \mathrm {k}}, \forall \mathrm {w} _ {\mathrm {j}, \mathrm {k}} \in \mathrm {L}, 1 \leq \mathrm {j} <   \mathrm {i} <   \mathrm {k} \leq \mathrm {n} \right\}, \\ \mathrm {E} \left(\mathrm {c} _ {\mathrm {i}}\right) = \left\{\mathrm {w} _ {\mathrm {j}, \mathrm {i}}, \forall \mathrm {w} _ {\mathrm {j}, \mathrm {i}} \in \mathrm {L}, 1 \leq \mathrm {j} <   \mathrm {i} \right\}, \\ \mathrm {S} \left(\mathrm {c} _ {\mathrm {i}}\right) = \left\{\mathrm {c} _ {\mathrm {i}}, \exists \mathrm {c} _ {\mathrm {i}} \in \mathrm {L} \right\}. \tag {8} \\ \end{array}
$$
Here, L denotes the lexicon we use in this work. Additionally, if a word set is empty, a special word "NONE" is added to the empty word set. An example of this categorization approach is shown in Figure 3. Noted that in this way, not only we
Figure 3: The SoftLexicon method.
can introduce the word embedding, but also no information loss exists since the matching results can be exactly restored from the four word sets of the characters.
Condensing the word sets. After obtaining the "BMES" word sets for each character, each word set is then condensed into a fixed-dimensional vector. In this work, we explored two approaches for implementing this condensation.
The first implementation is the intuitive mean-pooling method:
$$
\boldsymbol {v} ^ {s} (\mathcal {S}) = \frac {1}{| \mathcal {S} |} \sum_ {w \in \mathcal {S}} e ^ {w} (w). \tag {9}
$$
Here,  $S$  denotes a word set and  $e^w$  denotes the word embedding lookup table.
However, as shown in Table 8, the results of empirical studies revealed that this algorithm does not perform well. Therefore, a weighting algorithm is introduced to further leverage the word information. To maintain computational efficiency, we did not opt for a dynamic weighting algorithm like attention. Instead, we propose using the frequency of each word as an indication of its weight. Since the frequency of a word is a static value that can be obtained offline, this can greatly accelerate the calculation of the weight of each word.
Specifically, let  $z(w)$  denote the frequency that a lexicon word  $w$  occurs in the statistical data, the weighted representation of the word set  $S$  is obtained as follows:
$$
\boldsymbol {v} ^ {s} (S) = \frac {4}{Z} \sum_ {w \in S} z (w) \boldsymbol {e} ^ {w} (w), \tag {10}
$$
$$
Z = \sum_ {w \in \mathrm {B} \cup \mathrm {M} \cup \mathrm {E} \cup \mathrm {S}} z (w).
$$
5954
Here, weight normalization is performed on all words in the four word sets to make an overall comparison.
In this work, the statistical data set is constructed from a combination of training and developing data of the task. Of course, if there is unlabelled data in the task, the unlabeled data set can serve as the statistical data set. In addition, note that the frequency of  $w$  does not increase if  $w$  is covered by another sub-sequence that matches the lexicon. This prevents the problem in which the frequency of a shorter word is always less than the frequency of the longer word that covers it.
Combining with character representation. The final step is to combine the representations of four word sets into one fix-dimensional feature, and add it to the representation of each character. In order to retain as much information as possible, we choose to concatenate the representations of the four word sets, and the final representation of each character is obtained by:
$$
\boldsymbol {e} ^ {\mathrm {s}} (\mathrm {B}, \mathrm {M}, \mathrm {E}, \mathrm {S}) = \left[ \boldsymbol {v} ^ {\mathrm {s}} (\mathrm {B}); \boldsymbol {v} ^ {\mathrm {s}} (\mathrm {M}); \boldsymbol {v} ^ {\mathrm {s}} (\mathrm {E}); \boldsymbol {v} ^ {\mathrm {s}} (\mathrm {S}) \right], \tag {11}
$$
$$
x ^ {c} \leftarrow [ x ^ {c}; e ^ {s} (\mathrm {B}, \mathrm {M}, \mathrm {E}, \mathrm {S}) ].
$$
Here,  $\boldsymbol{v}^s$  denotes the weighting function above.
3.3 Sequence Modeling Layer
With the lexicon information incorporated, the character representations are then put into the sequence modeling layer, which models the dependency between characters. Generic architectures for this layer including the bidirectional long-short term memory network(BiLSTM), the Convolutional Neural Network(CNN) and the transformer(Vaswani et al., 2017). In this work, we implemented this layer with a single-layer BiLSTM.
Here, we precisely show the definition of the forward LSTM:
$$
\left[ \begin{array}{l} i _ {t} \\ f _ {t} \\ o _ {t} \\ \widetilde {c} _ {t} \end{array} \right] = \left[ \begin{array}{l} \sigma \\ \sigma \\ \sigma \\ \tanh  \end{array} \right] \left(W \left[ \begin{array}{l} x _ {t} ^ {c} \\ h _ {t - 1} \end{array} \right] + b\right), \tag {12}
$$
$$
c _ {t} = \widetilde {c} _ {t} \odot i _ {t} + c _ {t - 1} \odot f _ {t},
$$
$$
h _ {t} = o _ {t} \odot \tanh  (c _ {t}).
$$
where  $\sigma$  is the element-wise sigmoid function and  $\odot$  represents element-wise product. W and b are trainable parameters. The backward LSTM shares the same definition as the forward LSTM
Datasets	Type	Train	Dev	Test
OntoNotes	Sentence	15.7k	4.3k	4.3k
Char	491.9k	200.5k	208.1k
MSRA	Sentence	46.4k	-	4.4k
Char	2169.9k	-	172.6k
Weibo	Sentence	1.4k	0.27k	0.27k
Char	73.8k	14.5	14.8k
Resume	Sentence	3.8k	0.46	0.48k
Char	124.1k	13.9k	15.1k
yet model the sequence in a reverse order. The concatenated hidden states at the  $i^{th}$  step of the forward and backward LSTMs  $h_i = [\vec{h}_i; \vec{h}_i]$  forms the context-dependent representation of  $c_i$ .
3.4 Label Inference Layer
On top of the sequence modeling layer, it is typical to apply a sequential conditional random field (CRF) (Lafferty et al., 2001) layer to perform label inference for the whole character sequence at once:
$$
p (y | s; \boldsymbol {\theta}) = \frac {\prod_ {t = 1} ^ {n} \phi_ {t} \left(y _ {t - 1} , y _ {t} \mid s\right)}{\sum_ {y ^ {\prime} \in \mathcal {Y} _ {s}} \prod_ {t = 1} ^ {n} \phi_ {t} \left(y _ {t - 1} ^ {\prime} , y _ {t} ^ {\prime} \mid s\right)}. \tag {13}
$$
Here,  $\mathcal{V}_s$  denotes all possible label sequences of  $s$ , and  $\phi_t(y',y|s) = \exp (\pmb{w}_{y',y}^T h_t + b_{y',y})$ , where  $\pmb{w}_{y',y}$  and  $b_{y',y}$  are trainable parameters corresponding to the label pair  $(y',y)$ , and  $\pmb{\theta}$  denotes model parameters. For label inference, it searches for the label sequence  $y^{*}$  with the highest conditional probability given the input sequence  $s$ :
$$
y ^ {*} = _ {y} p (y | s; \boldsymbol {\theta}), \tag {14}
$$
which can be efficiently solved using the Viterbi algorithm (Forney, 1973).
Most experimental settings in this work followed the protocols of Lattice-LSTM (Zhang and Yang, 2018), including tested datasets, compared baselines, evaluation metrics (P, R, F1), and so on. To make this work self-completed, we concisely illustrate some primary settings of this work.
The methods were evaluated on four Chinese NER datasets, including OntoNotes (Weischedel et al., 2011), MSRA (Levow, 2006), Weibo NER (Peng
5955
Models	OntoNotes	MSRA	Weibo	Resume
Lattice-LSTM	1×	1×	1×	1×
LR-CNN (Gui et al., 2019)	2.23×	1.57×	2.41×	1.44×
BERT-tagsger	2.56×	2.55×	4.45×	3.12×
BERT + LSTM + CRF	2.77×	2.32×	2.84×	2.38×
SoftLexicon (LSTM)	6.15×	5.78×	6.10×	6.13×
SoftLexicon (LSTM) + bichar	6.08×	5.95×	5.91×	6.45×
SoftLexicon (LSTM) + BERT	2.74×	2.33×	2.85×	2.32×
Table 2: Inference speed (average sentences per second, the larger the better) of our method with LSTM layer compared with Lattice-LSTM, LR-CNN and BERT.
and Dredze, 2015; He and Sun, 2017a), and Resume NER (Zhang and Yang, 2018). OntoNotes and MSRA are from the newswire domain, where gold-standard segmentation is available for training data. For OntoNotes, gold segmentation is also available for development and testing data. Weibo NER and Resume NER are from social media and resume, respectively. There is no gold standard segmentation in these two datasets. Table 1 shows statistic information of these datasets. As for the lexicon, we used the same one as Lattice-LSTM, which contains 5.7k single-character words, 291.5k two-character words, 278.1k three-character words, and 129.1k other words. In addition, the pretrained character embeddings we used are also the same with Lattice-LSTM, which are pre-trained on Chinese Giga-Word using word2vec.
Implementation Detail
In this work, we implement the sequence-labeling layer with Bi-LSTM. Most implementation details followed those of Lattice-LSTM, including character and word embedding sizes, dropout, embedding initialization, and LSTM layer number. Additionally, the hidden size was set to 200 for small datasets Weibo and Resume, and 300 for larger datasets OntoNotes and MSRA. The initial learning rate was set to 0.005 for Weibo and 0.0015 for the rest three datasets with Adamax (Kingma and Ba, 2014) step rule  $^{2}$ .
4.2 Computational Efficiency Study
Table 2 shows the inference speed of the SoftLexicon method when implementing the sequence modeling layer with a bi-LSTM layer. The speed was evaluated based on the average number of sentences processed by the model per second using a GPU (NVIDIA TITAN X). From the
Figure 4: Inference speed against sentence length. We use a same batch size of 1 for a fair speed comparison.
table, we can observe that when decoding with the same batch size  $(= 1)$ , the proposed method is considerably more efficient than Lattice-LSTM and LR-CNN, performing up to 6.15 times faster than Lattice-LSTM. The inference speeds of SoftLexicon(LSTM) with bichar are close to those without bichar, since we only concatenate an additional feature to the character representation. The inference speeds of the BERT-Tagger and SoftLexicon (LSTM) + BERT models are limited due to the deep layers of the BERT structure. However, the speeds of the SoftLexicon (LSTM) + BERT model are still faster than those of Lattice-LSTM and LR-CNN on all datasets.