SlowGuess/NAACL_2022_OCR huggingface dataset:
Jinpeng Hu, Yaling Shen, Yang Liu Xiang Wan, Tsung-Hui Chang
Pazhou Lab, Guangzhou, 510330, China
{jinpenghu, yalingshen, yangliu5}@link.cuhk.edu.cn
wanxiang@sribd.cn changtsunghui@cuhk.edu.cn
Named entity recognition (NER) is a fundamental and important task in NLP, aiming at identifying named entities (NEs) from free text. Recently, since the multi-head attention mechanism applied in the Transformer model can effectively capture longer contextual information, Transformer-based models have become the mainstream methods and have achieved significant performance in this task. Unfortunately, although these models can capture effective global context information, they are still limited in the local feature and position information extraction, which is critical in NER. In this paper, to address this limitation, we propose a novel Hero-Gang Neural structure (HGN), including the Hero and Gang module, to leverage both global and local information to promote NER. Specifically, the Hero module is composed of a Transformer-based encoder to maintain the advantage of the self-attention mechanism, and the Gang module utilizes a multi-window recurrent module to extract local features and position information under the guidance of the Hero module. Afterward, the proposed multi-window attention effectively combines global information and multiple local features for predicting entity labels. Experimental results on several benchmark datasets demonstrate the effectiveness of our proposed model. $^{1}$
Named entity recognition (NER) is one of the most important and fundamental research topics in natural language processing (NLP), which recognizes named entities (NEs), such as person, location, disease from raw text. NER has attracted substantial attention in the past decades owing to its importance in downstream tasks, e.g., knowledge
graph construction (Bosselut et al., 2019), question-answering (Pergola et al., 2021), and relation extraction (He et al., 2019).
In the early stage, the popular methods for solving NER are some traditional machine learning methods, e.g., Hidden Markov Model (HMM) (Morwal et al., 2012) and conditional random field (CRF) (Mozharova and Loukachevitch, 2016), which require extracting features manually, making the process inefficient and time-consuming. With the breakthrough of recurrent neural networks (RNN) in NLP, Long short-term memory (LSTM) (Hochreiter et al., 1997) and Gated Recurrent Unit (GRU) (Cho et al., 2014) have become mainstream methods for this task and have achieved promising results since neural networks can automatically extract features from the sequence and also take each token's position information into consideration (Lample et al., 2016; Chiu and Nichols, 2016; Huang et al., 2015). Nevertheless, RNN fails to perform well with long sequences due to the gradients exploding and vanishing. In recent years, Transformer-based models (Vaswani et al., 2017) have become mainstream methods because these models are able to capture long-term dependencies with the help of multi-head attention and thus provide better global context information, especially for long sequences (Lee et al., 2020; Yang et al., 2019b). However, these Transformer-based models usually are insensitive to the local context since the representation of each token is computed by the canonical point-wise dot-product self-attention (Li et al., 2019; Huang et al., 2021). Besides, although some studies (Shaw et al., 2018; Devlin et al., 2018; Liu et al., 2019) have been proposed to inject position information into Transformer, they are still inadequate to help Transformer obtain appropriate position information (Huang et al., 2020; Qu et al., 2021). In other words, the self-attention mechanism is effective in overcoming the constraints of RNN from the perspective of long-sequence con
1Our code is released at https://github.com/jinpeng01/HGN.
Human Language Technologies, pages 1924 - 1936
Figure 1: The overall architecture of our proposed model. From left to right are the Hero module, Gang module, and multi-window attention, respectively, shown in different dashed boxes. The purple solid frame, green, and yellow dashed frames in the Hero module are sliding windows with different window sizes. The green box in the Gang module shows the multiple sub-sequences generated by the sliding windows for  $\mathbf{z}_4$ , and the grey box represents the bidirectional recurrent mechanism that is used to capture local features from these sub-sequences. Note that  $\overleftarrow{\mathbf{h}_1}$  and  $\overrightarrow{\mathbf{h}_{2k+1}}$  are the last hidden states of backward and forward recurrent structures. The extracted local information is shown in the yellow box with its corresponding sub-sequences in the green box.
text information extraction, but is inferior to RNN in terms of local contextual and position information extraction. Yet, both long-term dependencies and local context information are essential for the NER model to correctly identify entities.
Thus, to alleviate the shortcomings in RNN and Transformers while maintaining their respective strengths, in this paper, we propose a novel HeroGang Neural model to leverage both global and local contextual information to improve NER. In doing so, on the one hand, we utilize a Transformer-based sequence encoder (i.e., Hero module) to extract effective global contextual information with the help of the self-attention mechanism. On the other hand, a multi-window recurrent unit (i.e., Gang module) is applied to extract local features from multiple sub-sequences under the guidance of the extracted global information. Afterward, we propose to use multi-window attention to elaborately combine global and local contextual features. The performance of our proposed model significantly outperforms the strong baseline models on several NER benchmark datasets (including both general and biomedical domains) and achieves new state-of-the-art results on some datasets.
NER is usually performed as a sequence labeling problem. In detail, given a sequence of  $X = x_{1},x_{2},\ldots ,x_{N}$  with  $N$  tokens, we aim to
learn a function that maps the input sequence into another one with the corresponding label  $\hat{Y} = \hat{y}_1, \hat{y}_2, \hat{y}_3, \dots, \hat{y}_n$  in the same length. As summarized in Figure 1, the Transformer-based models (e.g., BERT (Devlin et al., 2018), XLNET (Yang et al., 2019b)) are regarded as the Hero module to model the entire sentence for global sequence information extraction and the Gang module is responsible for local and relative position information extraction. Afterward, we employ the multi-window attention to elaborately combine these different features (i.e., features extracted from the Hero and Gang modules), which is then used to predict labels for each token. Therefore, the aforementioned process can be formulated as:
$$
\hat {Y} = f (X, \mathrm {H} (X), \mathrm {G} (X)), \tag {1}
$$
where  $\mathrm{H}(\cdot)$  and  $\mathrm{G}(\cdot)$  refer to the Hero and Gang modules, respectively, and the details of them are presented in the following subsections.
2.1 Hero Module
The role of the Hero module in our proposed model is similar to that of the leader in a team, who is responsible for providing guidance, offering instructions, giving directions, and assigning sub-tasks to fellow memberships. Therefore, the Hero module is required to have a comprehensive understanding of the task, including overall and local progress. Thanks to the characteristics of the multi-head self-attention mechanism, Transformer is powerful in
modeling long sequences and can provide more effective global information than other counterpart models, and it has already achieved promising results in the NER task (Luo et al., 2020; Beltagy et al., 2019). Thus, we employ a Transformer-based encoder as our Hero module to obtain the global context information  $\mathbf{z}_i$  for each token  $x_i$  by
$$
[ \mathbf {z} _ {1}, \mathbf {z} _ {2}, \dots , \mathbf {z} _ {N} ] = f _ {\mathrm {H}} (x _ {1}, x _ {2}, \dots , x _ {N}). \tag {2}
$$
Herein,  $f_{\mathrm{H}}(\cdot)$  refers to a pre-trained Transformer-based sequence encoder (e.g., BERT (Devlin et al., 2018) and BioBERT (Lee et al., 2020)). The features  $\mathbf{z}$  are then input to the Gang module for extracting local contextual features and their corresponding relative position information.
2.2 Gang Module
As introduced in the previous section, although pretrained models are able to provide effective global contextual representation, it lacks the ability to extract local features and relative position information. Thus, we propose a multi-window recurrent module, named Gang, to enhance local information extraction. Recurrent structures (RS), such as LSTM, GRU, and tradition RNN are effective in extracting both local and relative position information from the sequence, owing to characteristics of the recurrent mechanism. To better emphasize the local features of each word without being disturbed by long-distance information, we construct a sliding window with a fixed length to generate shorter sub-sequences, where each sub-sequence includes several consecutive elements in  $\mathbf{z}$ . An additional advantage of this operation is that, in comparison with the whole sequence, the sub-sequence is much shorter so that it is easier to be modeled by the RS.
In detail, for a single sliding window with length  $k$ , each hidden state  $z_{i}$  from the Hero module, the corresponding sub-sequence is  $\mathbf{z}_{i-k}, \mathbf{z}_{i-k+1}, \ldots, \mathbf{z}_{i}, \ldots, \mathbf{z}_{i+k-1}, \mathbf{z}_{i+k}$  that includes  $2k + 1$  consecutive tokens. This sub-sequence of length  $2k + 1$  contains rich local contextual information of  $x_{i}$ , and thus we utilize an RS to encode it for obtaining local semantic and relative position information. To extract the local information of two directions, we utilize a bidirectional structure to encode this sequence span, where the forward RS computes a representation  $\overleftarrow{\mathbf{h}}_{2k+1}$  from left to right, and the other backward RS computes a vector  $\overleftarrow{\mathbf{h}}_{1}$  for the same sub-sequence in reverse. We concatenate the  $\overleftarrow{\mathbf{h}}_{1}$  and  $\overleftarrow{\mathbf{h}}_{2k+1}$  as the local feature
$\mathbf{h}_i = \overleftarrow{\left[\mathbf{h}_1,\mathbf{h}_{2k + 1}\right]}$  for token  $x_{i}$  , and then we can obtain local features for each token in sequence  $X$  via the similar way, denoted as  $\mathbf{h} = \mathbf{h}_1,\mathbf{h}_2,\dots ,\mathbf{h}_N$
In practice, we need to consider two situations. First, each token might have multiple levels of local information, such as phrase-level and clause-level, which may affect the understanding of the current token. Second, since different tokens or the same token in various contexts might have different relationships with their surrounding words, we need to consider more sub-sequences with varying lengths for obtaining more comprehensive local contextual information. Therefore, we propose to utilize multiple sliding windows with different window sizes to extract richer local features to alleviate the above issues. We assume that local features  $\mathbf{h}^1,\mathbf{h}^2,\dots ,\mathbf{h}^M$  are extracted from different groups of sub-sequences, whose corresponding window lengths are  $k^{1},k^{2},\dots ,k^{M}$ . This process can be formulated as:
$$
\mathbf {h} ^ {1}, \mathbf {h} ^ {2}, \dots , \mathbf {h} ^ {M} = \operatorname {G a n g} \left(k ^ {1}, k ^ {2}, \dots , k ^ {M}, \mathbf {z}\right), \tag {3}
$$
where  $M$  is the number of sliding windows and  $\mathbf{h}^j$  is a group of local features extracted from the corresponding sliding window with length  $k^j$ . The process is similar to the task assignment in the team, where different members are responsible for their own sub-tasks.
2.3 Multi-window Attention
We obtain global representation  $\mathbf{z}$  from the Hero module and multiple local features  $\mathbf{h}^1, \mathbf{h}^2, \dots, \mathbf{h}^M$  from the Gang module. Next, we apply the multi-window attention to effectively combine global contextual information and local features. In doing so, two types of attention methods are proposed in our model: MLP-Attention and DOT-Attention, respectively.
MLP-Attention We first concatenate these local features with global information and obtain the intermediate state  $\mathbf{m}$  by a fully connected layer.
$$
\mathbf {m} = \operatorname {M L P} ([ \mathbf {z}, \mathbf {H} ]), \tag {4}
$$
where  $\mathbf{H} = [\mathbf{h}^1, \mathbf{h}^2, \dots, \mathbf{h}^M]$  and  $\mathbf{m}$  have the same dimension as  $\mathbf{z}$ . MLP represents a fully connected layer. Then  $\mathbf{m}$  is used as a query vector and  $[\mathbf{z}, \mathbf{H}]$  serves as the key and value matrix. The final token representation can be computed by
$$
\mathbf {s} = \operatorname {s o f t m a x} (\mathbf {m} ([ \mathbf {z}, \mathbf {H} ]) ^ {\top}) [ \mathbf {z}, \mathbf {H} ]. \tag {5}
$$
DOT-Attention Instead of using a fully connected
Type	Dataset	TRAIN	VAL	TEST
#SENT.	#ENT.	#AS.	#SENT.	#ENT.	#AS.	#SENT.	#ENT.	#AS.
GENERAL	W16	2.4k	1.5k	19.41	1.0k	0.7k	16.26	3.9k	3.5k	16.08
W17	3.4k	2.0k	18.48	1.0k	0.8k	15.59	1.3k	1.1k	18.18
ON5E	59.9k	81.8k	18.17	8.5k	11.1k	17.32	8.3k	11.3k	18.49
BIOMED	BC5-D	4.6k	4.2k	25.79	4.6k	4.2k	25.52	4.8k	4.4k	25.92
BC2GM	12.6k	15.2k	28.14	2.5k	3.0k	28.07	5.0k	6.3k	28.33
BC5-C	4.6k	4.2k	25.79	4.6k	4.2k	25.52	4.8k	4.4k	25.92
Table 1: The statistics of the six benchmark datasets w.r.t. their training, validation and test sets, including the number of sentences (#Sent.), the number of entities (#Ent.), and the averaged word-based length (#AS.).
layer to generate a query vector, in this approach, we directly regard  $\mathbf{z}$  as the query vector and  $\mathbf{H}$  as the key and value matrix. We can obtain the final local feature by
$$
\mathbf {u} = \operatorname {s o f t m a x} (\mathbf {z} (\mathbf {H}) ^ {\top}) \mathbf {H}. \tag {6}
$$
Since  $\mathbf{u}$  is a weighted sum of different local features without considering global information, we use the sum of  $\mathbf{u}_i$  and  $\mathbf{z}_i$  as the final representation for each token  $x_i$ . Thus, the final representation can be obtained by
$$
\mathbf {s} = \left\{\mathbf {z} _ {1} + \mathbf {u} _ {1}, \mathbf {z} _ {2} + \mathbf {u} _ {2}, \dots , \mathbf {z} _ {N} + \mathbf {u} _ {N} \right\}. \tag {7}
$$
After obtaining the final representation from MLP-Attention or DOT-Attention, s is sent to the corresponding classifier implemented by the softmax function to predict the distribution of labels for each token in  $X$ .
3 Experiments Settings
In our experiments, six datasets are used in our experiments, WNUT17 (W17) (Strauss et al., 2016), WNUT16 (W16) (Derczynski et al., 2017), OntoNotes 5.0 (ON5e) (Pradhan et al., 2013), BC5CDR-disease (BC5-D), BC2GM, and BC5CDR-chem (BC5-C). The W17 and W16 are social media benchmark datasets constructed from Twitter, and ON5e is a general domain dataset consisting of diverse sources like telephone conversations, newswire, etc. BC5CDR, including both BC5-D and BC5-C, is a dataset used for the BioCreative V Chemical Disease Relation Task and contains chemical and disease mentions, where humans manually annotate the annotations. BC2GM is the dataset that is usually utilized for the BioCreative II gene mention tagging task and contains 20000 sentences from the abstracts of biomedical publications. For all datasets, we utilize the official splits for a fair evaluation and the statistics of the
datasets are shown in Table 1. Besides, we follow previous studies that the final models are trained on training and validation sets on each dataset except the ON5e dataset.
For metrics, we exploit the same evaluation metrics used by previous works where precision (P), recall (R), and F-1 score are reported to evaluate the performance of our model.
We implement our model based on transformers (Wolf et al., 2020) $^{2}$  and employ pre-trained models to obtain global contextualized representation. Specifically, for general domain datasets (i.e., W16, W17 and ON5e), we use BERT-based-large (Devlin et al., 2018) $^{3}$  and XLNET-large-cased (Yang et al., 2019b) $^{4}$  as our Hero module. For biomedical datasets, BioBERT (Lee et al., 2020) $^{5}$  is utilized to obtain global information. We follow their default settings for all BERT, XLNET, and BioBERT: 24 layers of self-attention with 1024 dimensional embeddings. For hyperparameters of the Gang module, the hidden sizes of bidirectional recurrent structures for each window size are half of the embedding dimension from the output of the Hero module (i.e., 512). During the training process, we use Adam (Kingma and Ba, 2014) to optimize the negative log-likelihood loss function. More training details are shown in the Appendix A.1. Besides, we also compare four operations to combine different level features from the Hero and Gang module: MLP-Attention, DOT-Attention, concatenation, and summation, respectively, where concatenation is to connect all features directly through
2https://github.com/huggingface/ transformers
3We obtain the pre-trained BERT from https:// github.com/google-research/bert.
4We obtain XLNET from https://github.com/zihangdai/xlnet.
<sup>5</sup>We obtain BioBERT from https://github.com/dmis-lab/biobert
Methods	W16	W17	ON5E
P	R	F-1	P	R	F-1	P	R	F-1
with incorporating extra resources
SANER (Nie et al., 2020b)	-	51.27	55.01	-	49.45	50.36	-	-	-
AESUBER (Nie et al., 2020a)	-	-	55.14	-	-	50.68	-	-	90.32
HIRE-NER (Luo et al., 2020)	-	-	-	-	-	-	-	-	90.30
CL-KL (Wang et al., 2021)	-	-	58.98	-	-	60.45	-	-	-
SYN-LSTM-CRF (Xu et al., 2021)	-	-	-	-	-	-	90.14	91.58	90.85
without extra resources
CNN-BILSTM-CRF (Chiu and Nichols, 2016)	-	-	-	-	-	-	86.04	86.53	86.28
BERT (Devlin et al., 2018)	-	49.02	54.36	-	46.73	49.52	-	-	89.16
XLNET (Yang et al., 2019b)	55.94	57.46	56.69	58.68	49.18	53.51	89.72	91.05	90.38
ASTRA (Wang et al., 2020)	-	-	-	-	-	49.72	-	-	89.44
BARTNER (Yan et al., 2021)	-	-	-	-	-	-	89.99	90.77	90.38
HGN (BERT) (CONCAT)	56.06	55.61	55.84	57.41	45.45	50.74	89.20	89.85	89.52
HGN (BERT) (ADD)	54.63	55.38	55.01	58.46	45.55	51.20	89.16	90.01	89.58
HGN (BERT) (MLP)	57.72	55.66	56.67	59.26	50.70	54.65	89.19	90.24	89.71
HGN (BERT) (DOT)	57.51	56.00	56.75	60.09	48.29	53.55	89.32	90.11	89.71
HGN (XLNET) (CONCAT)	57.48	57.90	57.69	63.39	49.27	55.45	89.92	91.35	90.63
HGN (XLNET) (ADD)	57.31	58.05	57.68	59.11	48.36	53.20	90.10	91.39	90.74
HGN (XLNET) (MLP)	58.91	59.89	59.39	63.16	52.27	57.20	90.29	91.56	90.92
HGN (XLNET) (DOT)	59.74	59.26	59.50	62.49	53.10	57.41	90.10	91.64	90.86
Table 2: Comparisons of our proposed models with previous studies on the W16, W17, and ON5e, respectively, with respect to precision, recall, and F-1 score for NER. Previous studies are divided into two parts from top to bottom, representing methods requiring extra resources and without such requirements, respectively.
$\mathbf{s} = [\mathbf{h}^1, \mathbf{h}^2, \dots, \mathbf{h}^M, \mathbf{z}]$ , and summation is to add up these features by  $\mathbf{s} = \mathbf{h}^1 + \mathbf{h}^2 + \dots + \mathbf{h}^M + \mathbf{z}$ .
To explore the impact of our proposed model, we compare our model to the previous studies. For general domain, following baselines are compared in our experiment on W16, W17 and ON5e.
- CNN-BiLSTM-CRF (Chiu and Nichols, 2016) utilizes a hybrid bidirectional and CNN architecture to detect word-and character-level features.
- BERT (Devlin et al., 2018) is a pre-trained language model and we apply it to the NER task by direct fine-tuning.
- SANER (Nie et al., 2020b), CL-KL (Wang et al., 2021) and AESUBER (Nie et al., 2020a) improve entity recognition by leveraging syntactic information or semantically relevant texts.
- HIRE-NER (Luo et al., 2020) utilizes both sentence-level and document-level representations to improve sequence labeling.
- SYN-LSTM-CRF (Xu et al., 2021) integrates the structured information by graph-encoded representations obtained from GNNs.
- BARTNER (Yan et al., 2021) formulates NER tasks as a span sequence generation problem.
In addition, we also compare our proposed model with the following baselines on the aforementioned biomedical datasets:
- MTM-CW (Wang et al., 2019a), BILM
(Sachan et al., 2018), NCBI_BERT (Peng et al., 2019), MT-BIONER (Tong et al., 2021) utilize multi-task learning or transfer learning to enhance biomedical NER.
- BIOBERT (Lee et al., 2020) is a pre-trained model trained with a large amount of biomedical corpus and then applied by directly fine-tuning.
- KEBIO-LM (Yuan et al., 2021) proposes a biomedical pre-trained language model that incorporates knowledge from the Unified Medical Language System (UMLS).
Note that in both general and biomedical domains, our model does not require external resources.
4.1 General Domain NER
In this subsection, to explore the effectiveness of our proposed model, we conduct experiments to compare our model with existing studies, and the results are reported in Table 2. There are several observations drawn from different aspects. First, when we make a fair comparison without extra resources (e.g., BERT, XLNET, and ASTRA), our model obtains significant improvements on all datasets in terms of Precision, Recall, and F1, which confirms the effectiveness of our proposed Hero-Gang neural structure. This is because multiple-level features can be reasonably encoded into the model and thus alleviate the limitations of
Methods	BC5-D	BC2GM	BC5-C
P	R	F-1	P	R	F-1	P	R	F-1
with incorporating extra resources
BiLM (Sachan et al., 2018)	-	-	-	81.81	81.57	81.69	-	-	-
MTM-CW (Wang et al., 2019a)	-	-	-	82.10	79.42	80.74	-	-	-
KEBIO-LM (Yuan et al., 2021)	-	-	86.10	-	-	85.10	-	-	93.30
MT-BIONER (Tong et al., 2021)	-	-	-	84.42	85.14	84.78	93.29	94.69	93.98
without extra resources
NCBI_BERT (Peng et al., 2019)	-	-	86.60	-	-	-	-	-	93.50
BIOBERT (Lee et al., 2020)	86.47	87.84	87.15	84.32	85.12	84.72	93.68	93.26	93.47
HGN (BIOBERT) (CONCAT)	85.90	88.81	87.33	83.91	86.36	85.12	94.30	93.93	94.11
HGN (BIOBERT) (ADD)	85.89	88.74	87.29	85.21	85.50	85.35	94.01	94.57	94.29
HGN (BIOBERT) (MLP)	86.70	88.86	87.77	84.93	86.37	85.65	94.23	94.63	94.43
HGN (BIOBERT) (DOT)	86.27	89.51	87.86	85.21	85.88	85.54	94.45	94.73	94.59
Table 3: Comparisons of our proposed models with previous studies on the BC5-D, BC2GM, and BC5-C, respectively, for biomedical NER in terms of precision, recall, and F-1 score. Previous works are divided into two sections, indicating methods requiring extra resources and without such requirements.
Transformer in local feature extraction. Second, although some complicated models enhance NER by incorporating extra knowledge, e.g., SANER uses augmented semantic information, Hire-NER utilizes two-level hierarchical contextualized representations, and CL-KL selects a set of semantically relevant texts to improve NER, our model achieves competitive results without such requirements. This is because each word in the natural text usually has a closer relationship with its surrounding words, especially the adjacent words, such that features extracted by the Gang module can provide more valuable information for NER, and thus our model achieves promising performance. Third, the XLNET-based model obtains better results than the BERT-based model, which indicates that XLNET can generate more effective representations on the NER task. The reason behind this might be that XLNET combines the permutation operation with the autoregressive technology to further improve representation learning, so that XLNET can provide a better text understanding than BERT.
4.2 Biomedical NER
We also compare our model with state-of-the-art models in the biomedical NER on the aforementioned datasets with all results reported in Table 3. There are several observations. First, we can see that our model outperforms existing methods, regardless of whether they introduce external knowledge, which further confirms the validity of our innovation in combining local and global features to enhance feature extraction. Second, although some models utilize higher-level features, e.g., BIOKM-NER leverages POS labels, syntactic constituents,