Venue: NAACL 2022

# Hero-Gang Neural Model For Named Entity Recognition

Jinpeng Hu\({}^{\heartsuit}\), **Yaling Shen\({}^{\heartsuit}\),****Yang Liu\({}^{\heartsuit}\)**

Xiang Wan\({}^{\heartsuit\diamond\dagger}\),****Tsung-Hui Chang\({}^{\heartsuit\dagger}\)

\({}^{\heartsuit}\)Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong,

Shenzhen, Guangdong, China

\({}^{\heartsuit}\)Pazhou Lab, Guangzhou, 510330, China

{jinpenghu, yalingshen, yangliu5}@link.cuhk.edu.cn

wanxiang@sribd.cn changtsunghui@cuhk.edu.cn

###### Abstract

Named entity recognition (NER) is a fundamental and important task in NLP, aiming at identifying named entities (NEs) from free text. Recently, since the multi-head attention mechanism applied in the Transformer model can effectively capture longer contextual information, Transformer-based models have become the mainstream methods and have achieved significant performance in this task. Unfortunately, although these models can capture effective global context information, they are still limited in the local feature and position information extraction, which is critical in NER. In this paper, to address this limitation, we propose a novel Hero-Gang Neural structure (HGN), including the Hero and Gang module, to leverage both global and local information to promote NER. Specifically, the Hero module is composed of a Transformer-based encoder to maintain the advantage of the self-attention mechanism, and the Gang module utilizes a multi-window recurrent module to extract local features and position information under the guidance of the Hero module. Afterward, the proposed multi-window attention effectively combines global information and multiple local features for predicting entity labels. Experimental results on several benchmark datasets demonstrate the effectiveness of our proposed model.1

Footnote 1: Corresponding author.

+
Footnote †: Our code is released at [https://github.com/jinpeng01/HGN](https://github.com/jinpeng01/HGN).

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1924 - 1936

July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Named entity recognition (NER) is one of the most important and fundamental research topics in natural language processing (NLP), which recognizes named entities (NEs), such as person, location, disease from raw text. NER has attracted substantial attention in the past decades owing to its importance in downstream tasks, e.g., knowledge graph construction Bosselut et al. (2019), question-answering Pergola et al. (2021), and relation extraction He et al. (2019).

In the early stage, the popular methods for solving NER are some traditional machine learning methods, e.g., Hidden Markov Model (HMM) Morwal et al. (2012) and conditional random field (CRF) Mozharova and Loukachevitch (2016), which require extracting features manually, making the process inefficient and time-consuming. With the breakthrough of recurrent neural networks (RNN) in NLP, Long short-term memory (LSTM) Hochreiter et al. (1997) and Gated Recurrent Unit (GRU) Cho et al. (2014) have become mainstream methods for this task and have achieved promising results since neural networks can automatically extract features from the sequence and also take each token's position information into consideration Lample et al. (2016); Chiu and Nichols (2016); Huang et al. (2015). Nevertheless, RNN fails to perform well with long sequences due to the gradients exploding and vanishing. In recent years, Transformer-based models Vaswani et al. (2017) have become mainstream methods because these models are able to capture long-term dependencies with the help of multi-head attention and thus provide better global context information, especially for long sequences Lee et al. (2020); Yang et al. (2019b). However, these Transformer-based models usually are insensitive to the local context since the representation of each token is computed by the canonical point-wise dot-product self-attention Li et al. (2019); Huang et al. (2021). Besides, although some studies Shaw et al. (2018); Devlin et al. (2018); Liu et al. (2019) have been proposed to inject position information into Transformer, they are still inadequate to help Transformer obtain appropriate position information Huang et al. (2020); Qu et al. (2021). In other words, the self-attention mechanism is effective in overcoming the constraints of RNN from the perspective of long-sequence con

text information extraction, but is inferior to RNN in terms of local contextual and position information extraction. Yet, both long-term dependencies and local context information are essential for the NER model to correctly identify entities.

Thus, to alleviate the shortcomings in RNN and Transformers while maintaining their respective strengths, in this paper, we propose a novel Hero-Gang Neural model to leverage both global and local contextual information to improve NER. In doing so, on the one hand, we utilize a Transformer-based sequence encoder (i.e., Hero module) to extract effective global contextual information with the help of the self-attention mechanism. On the other hand, a multi-window recurrent unit (i.e., Gang module) is applied to extract local features from multiple sub-sequences under the guidance of the extracted global information. Afterward, we propose to use multi-window attention to elaborately combine global and local contextual features. The performance of our proposed model significantly outperforms the strong baseline models on several NER benchmark datasets (including both general and biomedical domains) and achieves new state-of-the-art results on some datasets.

## 2 Method

NER is usually performed as a sequence labeling problem. In detail, given a sequence of \(X=x_{1},x_{2},...,x_{N}\) with \(N\) tokens, we aim to learn a function that maps the input sequence into another one with the corresponding label \(\hat{Y}=\hat{y_{1}},\hat{y_{2}},\hat{y_{3}},...,\hat{y_{n}}\) in the same length. As summarized in Figure 1, the Transformer-based models (e.g., BERT Devlin et al. (2018), XLNET Yang et al. (2019b)) are regarded as the Hero module to model the entire sentence for global sequence information extraction and the Gang module is responsible for local and relative position information extraction. Afterward, we employ the multi-window attention to elaborately combine these different features (i.e., features extracted from the Hero and Gang modules), which is then used to predict labels for each token. Therefore, the aforementioned process can be formulated as:

\[\hat{Y}=f(X,\text{H}(X),\text{G}(X)), \tag{1}\]

where H(\(\cdot\)) and G(\(\cdot\)) refer to the Hero and Gang modules, respectively, and the details of them are presented in the following subsections.

### Hero Module

The role of the Hero module in our proposed model is similar to that of the leader in a team, who is responsible for providing guidance, offering instructions, giving directions, and assigning sub-tasks to fellow memberships. Therefore, the Hero module is required to have a comprehensive understanding of the task, including overall and local progress. Thanks to the characteristics of the multi-head self-attention mechanism, Transformer is powerful in

Figure 1: The overall architecture of our proposed model. From left to right are the Hero module, Gang module, and multi-window attention, respectively, shown in different dashed boxes. The purple solid frame, green, and yellow dashed frames in the Hero module are sliding windows with different window sizes. The green box in the Gang module shows the multiple sub-sequences generated by the sliding windows for \(\mathbf{z_{4}}\), and the grey box represents the bidirectional recurrent mechanism that is used to capture local features from these sub-sequences. Note that \(\overleftarrow{\mathbf{h_{1}}}\) and \(\overrightarrow{\mathbf{h_{2k+1}}}\) are the last hidden states of backward and forward recurrent structures. The extracted local information is shown in the yellow box with its corresponding sub-sequences in the green box.

 

modeling long sequences and can provide more effective global information than other counterpart models, and it has already achieved promising results in the NER task Luo et al. (2020); Beltagy et al. (2019). Thus, we employ a Transformer-based encoder as our Hero module to obtain the global context information \({\bf z}_{i}\) for each token \(x_{i}\) by

\[[{\bf z}_{1},{\bf z}_{2},\cdots,{\bf z}_{N}]=f_{\rm H}(x_{1},x_{2},...,x_{N}). \tag{2}\]

Herein, \(f_{\rm H}(\cdot)\) refers to a pre-trained Transformer-based sequence encoder (e.g., BERT Devlin et al. (2018) and BioBERT Lee et al. (2020)). The features \({\bf z}\) are then input to the Gang module for extracting local contextual features and their corresponding relative position information.

### Gang Module

As introduced in the previous section, although pretrained models are able to provide effective global contextual representation, it lacks the ability to extract local features and relative position information. Thus, we propose a multi-window recurrent module, named Gang, to enhance local information extraction. Recurrent structures (RS), such as LSTM, GRU, and tradition RNN are effective in extracting both local and relative position information from the sequence, owing to characteristics of the recurrent mechanism. To better emphasize the local features of each word without being disturbed by long-distance information, we construct a sliding window with a fixed length to generate shorter sub-sequences, where each sub-sequence includes several consecutive elements in \({\bf z}\). An additional advantage of this operation is that, in comparison with the whole sequence, the sub-sequence is much shorter so that it is easier to be modeled by the RS.

In detail, for a single sliding window with length \(k\), each hidden state \(z_{i}\) from the Hero module, the corresponding sub-sequence is \({\bf z}_{i-k},{\bf z}_{i-k+1},...,{\bf z}_{i},...,{\bf z}_{i+k-1},{\bf z}_{i+ k}\) that includes \(2k+1\) consecutive tokens. This sub-sequence of length \(2k+1\) contains rich local contextual information of \(x_{i}\), and thus we utilize an RS to encode it for obtaining local semantic and relative position information. To extract the local information of two directions, we utilize a bidirectional structure to encode this sequence span, where the forward RS computes a representation \(\overrightarrow{{\bf h}_{2k+1}}\) from left to right, and the other backward RS computes a vector \(\overleftarrow{{\bf h}_{1}}\) for the same sub-sequence in reverse. We concatenate the \(\overleftarrow{{\bf h}_{1}}\) and \(\overrightarrow{{\bf h}_{2k+1}}\) as the local feature \({\bf h}_{i}=[\overleftarrow{{\bf h}_{1}},\overrightarrow{{\bf h}_{2k+1}}]\) for token \(x_{i}\), and then we can obtain local features for each token in sequence \(X\) via the similar way, denoted as \({\bf h}={\bf h}_{1},{\bf h}_{2},\cdots,{\bf h}_{N}\).

In practice, we need to consider two situations. First, each token might have multiple levels of local information, such as phrase-level and clause-level, which may affect the understanding of the current token. Second, since different tokens or the same token in various contexts might have different relationships with their surrounding words, we need to consider more sub-sequences with varying lengths for obtaining more comprehensive local contextual information. Therefore, we propose to utilize multiple sliding windows with different window sizes to extract richer local features to alleviate the above issues. We assume that local features \({\bf h^{1}},{\bf h^{2}},\cdots,{\bf h^{M}}\) are extracted from different groups of sub-sequences, whose corresponding window lengths are \(k^{1},k^{2},\cdots,k^{M}\). This process can be formulated as:

\[{\bf h^{1}},{\bf h^{2}},\cdots,{\bf h^{M}}=\text{Gang}(k^{1},k^{2},\cdots,k^{ M},{\bf z}), \tag{3}\]

where \(M\) is the number of sliding windows and \({\bf h^{j}}\) is a group of local features extracted from the corresponding sliding window with length \(k^{j}\). The process is similar to the task assignment in the team, where different members are responsible for their own sub-tasks.

### Multi-window Attention

We obtain global representation \({\bf z}\) from the Hero module and multiple local features \({\bf h^{1}},{\bf h^{2}},\cdots,{\bf h^{M}}\) from the Gang module. Next, we apply the multi-window attention to effectively combine global contextual information and local features. In doing so, two types of attention methods are proposed in our model: MLP-Attention and DOT-Attention, respectively.

#### MLP-Attention

We first concatenate these local features with global information and obtain the intermediate state \({\bf m}\) by a fully connected layer.

\[{\bf m}=\text{MLP}([{\bf z},{\bf H}]), \tag{4}\]

where \({\bf H}=[{\bf h^{1}},{\bf h^{2}},\cdots,{\bf h^{M}}]\) and \({\bf m}\) have the same dimension as \({\bf z}\). MLP represents a fully connected layer. Then \({\bf m}\) is used as a query vector and \([{\bf z},{\bf H}]\) serves as the key and value matrix. The final token representation can be computed by

\[{\bf s}=\text{softmax}({\bf m}([{\bf z},{\bf H}])^{\top})[{\bf z},{\bf H}]. \tag{5}\]

#### Dot-Attention

Instead of using a fully connected 

layer to generate a query vector, in this approach, we directly regard \({\bf z}\) as the query vector and \({\bf H}\) as the key and value matrix. We can obtain the final local feature by

\[{\bf u}=\text{softmax}({\bf z}({\bf H})^{\top}){\bf H}. \tag{6}\]

Since \({\bf u}\) is a weighted sum of different local features without considering global information, we use the sum of \({\bf u}_{i}\) and \({\bf z}_{i}\) as the final representation for each token \(x_{i}\). Thus, the final representation can be obtained by

\[{\bf s}=\{{\bf z}_{1}+{\bf u}_{1},{\bf z}_{2}+{\bf u}_{2},\cdots,{\bf z}_{N}+{ \bf u}_{N}\}. \tag{7}\]

After obtaining the final representation from MLP-Attention or DOT-Attention, \({\bf s}\) is sent to the corresponding classifier implemented by the softmax function to predict the distribution of labels for each token in \(X\).

## 3 Experiments Settings

### Dataset and Metrics

In our experiments, six datasets are used in our experiments, WNUT17 (W17) (Strauss et al., 2016), WNUT16 (W16) (Derczynski et al., 2017), OntoNotes 5.0 (ON5e) (Pradhan et al., 2013), BC5CDR-disease (BC5-D), BC2GM, and BC5CDR-chem (BC5-C). The W17 and W16 are social media benchmark datasets constructed from Twitter, and ON5e is a general domain dataset consisting of diverse sources like telephone conversations, newswire, etc. BC5CDR, including both BC5-D and BC5-C, is a dataset used for the BioCreative V Chemical Disease Relation Task and contains chemical and disease mentions, where humans manually annotate the annotations. BC2GM is the dataset that is usually utilized for the BioCreative II gene mention tagging task and contains 20000 sentences from the abstracts of biomedical publications. For all datasets, we utilize the official splits for a fair evaluation and the statistics of the datasets are shown in Table 1. Besides, we follow previous studies that the final models are trained on training and validation sets on each dataset except the ON5e dataset.

For metrics, we exploit the same evaluation metrics used by previous works where precision (P), recall (R), and F-1 score are reported to evaluate the performance of our model.

### Implementation Details

We implement our model based on transformers (Wolf et al., 2020)2 and employ pre-trained models to obtain global contextualized representation. Specifically, for general domain datasets (i.e., W16, W17 and ON5e), we use BERT-cased-large (Devlin et al., 2018)3 and XLNET-large-cased (Yang et al., 2019b)4 as our Hero module. For biomedical datasets, BioBERT (Lee et al., 2020)5 is utilized to obtain global information. We follow their default settings for all BERT, XLNET, and BioBERT: 24 layers of self-attention with 1024 dimensional embeddings. For hyperparameters of the Gang module, the hidden sizes of bidirectional recurrent structures for each window size are half of the embedding dimension from the output of the Hero module (i.e., 512). During the training process, we use Adam (Kingma and Ba, 2014) to optimize the negative log-likelihood loss function. More training details are shown in the Appendix A.1. Besides, we also compare four operations to combine different level features from the Hero and Gang module: MLP-Attention, DOT-Attention, concatenation, and summation, respectively, where concatenation is to connect all features directly through

\begin{table}
\begin{tabular}{l|l|r r r|r r r|r r} \hline \hline \multirow{2}{*}{**Type**} & \multirow{2}{*}{**Dataset**} & \multicolumn{2}{c|}{Train} & \multicolumn{2}{c|}{Val} & \multicolumn{2}{c}{Test} \\ \cline{3-10}  & & \#Sent. & \#Ent. & \#AS. & \#Sent. & \#Ent. & \#AS. & \#Sent. & \#Ent. & \#AS. \\ \hline \multirow{3}{*}{General} & W16 & 2.4k & 1.5k & 19.41 & 1.0k & 0.7k & 16.26 & 3.9k & 3.5k & 16.08 \\  & W17 & 3.4k & 2.0k & 18.48 & 1.0k & 0.8k & 15.59 & 1.3k & 1.1k & 18.18 \\  & ONSée & 59.9k & 81.8k & 18.17 & 8.5k & 11.1k & 17.32 & 8.3k & 11.3k & 18.49 \\ \hline \multirow{3}{*}{Biomed} & BC5-D & 4.6k & 4.2k & 25.79 & 4.6k & 4.2k & 25.52 & 4.8k & 4.4k & 25.92 \\  & BC2GM & 12.6k & 15.2k & 28.14 & 2.5k & 3.0k & 28.07 & 5.0k & 6.3k & 28.33 \\ \cline{1-1}  & BC5-C & 4.6k & 4.2k & 25.79 & 4.6k & 4.2k & 25.52 & 4.8k & 4.4k & 25.92 \\ \hline \hline \end{tabular}
\end{table}
Table 1: The statistics of the six benchmark datasets w.r.t. their training, validation and test sets, including the number of sentences (#Sent.), the number of entities (#Ent.), and the averaged word-based length (#AS.).

 

\({\bf s}=[{\bf h}^{1},{\bf h}^{2},\cdots,{\bf h}^{M},{\bf z}]\), and summation is to add up these features by \({\bf s}={\bf h}^{1}+{\bf h}^{2}+\cdots+{\bf h}^{M}+{\bf z}\).

### Baselines

To explore the impact of our proposed model, we compare our model to the previous studies. For general domain, following baselines are compared in our experiment on W16, W17 and ON5e.

* **CNN-BiLSTM-CRF**Chiu and Nichols (2016) utilizes a hybrid bidirectional and CNN architecture to detect word-and character-level features.
* **BERT**Devlin et al. (2018) is a pre-trained language model and we apply it to the NER task by direct fine-tuning.
* **SANER**Nie et al. (2020), **CL-KL**Wang et al. (2021) and **AESUBER**Nie et al. (2020) improve entity recognition by leveraging syntactic information or semantically relevant texts.
* **Hire-NER**Luo et al. (2020) utilizes both sentence-level and document-level representations to improve sequence labeling.
* **Syn-LSTM-CRF**Xu et al. (2021) integrates the structured information by graph-encoded representations obtained from GNNs.
* **BARTNER**Yan et al. (2021) formulates NER tasks as a span sequence generation problem.

In addition, we also compare our proposed model with the following baselines on the aforementioned biomedical datasets:

* **MTM-CW**Wang et al. (2019), **BiLM**Sachan et al. (2018), **NCBI_BERT**Peng et al. (2019), **MT-BioNER**Tong et al. (2021) utilize multi-task learning or transfer learning to enhance biomedical NER.
* **BioBERT**Lee et al. (2020) is a pre-trained model trained with a large amount of biomedical corpus and then applied by directly fine-tuning.
* **KeBio-LM**Yuan et al. (2021) proposes a biomedical pre-trained language model that incorporates knowledge from the Unified Medical Language System (UMLS). Note that in both general and biomedical domains, our model **does not require external resources.**

## 4 Results and Analyses

### General Domain NER

In this subsection, to explore the effectiveness of our proposed model, we conduct experiments to compare our model with existing studies, and the results are reported in Table 2. There are several observations drawn from different aspects. First, when we make a fair comparison without extra resources (e.g., BERT, XLNET, and ASTRA), our model obtains significant improvements on all datasets in terms of Precision, Recall, and F-1, which confirms the effectiveness of our proposed Hero-Gang neural structure. This is because multiple-level features can be reasonably encoded into the model and thus alleviate the limitations of

\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline \multirow{2}{*}{**Methods**} & \multicolumn{3}{c}{**W16**} & \multicolumn{3}{c}{**W17**} & \multicolumn{3}{c}{**ON5e**} \\ \cline{2-10}  & **P** & **R** & **F-1** & **P** & **R** & **F-1** & **P** & **R** & **F-1** \\ \hline \multicolumn{10}{l}{**with incorporating extra resources**} \\ SANER Nie et al. (2020) & - & 51.27 & 55.01 & - & 49.45 & 50.36 & - & - & - \\ AESUBER Nie et al. (2020) & - & - & 55.14 & - & - & 50.68 & - & - & 90.32 \\ HIR-NER Luo et al. (2020) & - & - & - & - & - & - & - & - & 90.30 \\ CL-KL Wang et al. (2021) & - & - & **58.98** & - & - & **60.45** & - & - & - \\ Syn-LSTM-CRF Xu et al. (2021) & - & - & - & - & - & - & 90.14 & 91.58 & **90.85** \\ \hline \multicolumn{10}{l}{**without extra resources**} \\ CNN-BILSTM-CRF Chiu and Nichols (2016) & - & - & - & - & - & - & 86.04 & 86.53 & 86.28 \\ BERT Devlin et al. (2018) & - & 49.02 & 54.36 & - & 46.73 & 49.52 & - & - & 89.16 \\ XLNET Yang et al. (2019) & 55.94 & 57.46 & **56.69** & 58.68 & 49.18 & **53.51** & 89.72 & 91.05 & **90.38** \\ ASTRA Wang et al. (2020) & - & - & - & - & - & 49.72 & - & - & 89.44 \\ BARTNER Yan et al. (2021) & - & - & - & - & - & - & 89.99 & 90.77 & **90.38** \\ \hline HGN (BERT) (concat) & 56.06 & 55.61 & 55.84 & 57.41 & 45.45 & 50.74 & 89.20 & 89.85 & 89.52 \\ HGN (BERT) (add) & 54.63 & 55.38 & 55.01 & 58.46 & 45.55 & 51.20 & 89.16 & 90.01 & 99.58 \\ HGN (BERT) (MLP) & 57.72 & 55.66 & 56.67 & 59.26 & 50.70 & 54.65 & 89.19 & 90.24 & 89.71 \\ HGN (BERT) (dot) & 57.51 & 56.00 & 56.75 & 60.09 & 48.29 & 53.55 & 89.32 & 90.11 & 89.71 \\ HGN (XLNET) (concat) & 57.48 & 57.90 & 57.69 & 63.39 & 49.27 & 55.45 & 89.92 & 91.35 & 90.63 \\ HGN (XLNET) (add) & 57.31 & 58.05 & 57.68 & 59.11 & 48.36 & 53.20 & 90.10 & 91.39 & 90.74 \\ HGN (XLNET) (ddp) & 58.91 & 59.89 & 59.39 & 63.16 & 52.27 & 57.20 & 90.29 & 91.56 & **90.92** \\ HGN (XLNET) (dot) & 59.74 & 59.26 & **59.50** & 62.49 & 53.10 & **57.41** & 90.10 & 91.64 & 90.86 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparisons of our proposed models with previous studies on the W16, W17, and ON5e, respectively, with respect to precision, recall, and F-1 score for NER. Previous studies are divided into two parts from top to bottom, representing methods requiring extra resources and without such requirements, respectively.

 

Transformer in local feature extraction. Second, although some complicated models enhance NER by incorporating extra knowledge, e.g., SANER uses augmented semantic information, Hire-NER utilizes two-level hierarchical contextualized representations, and CL-KL selects a set of semantically relevant texts to improve NER, our model achieves competitive results without such requirements. This is because each word in the natural text usually has a closer relationship with its surrounding words, especially the adjacent words, such that features extracted by the Gang module can provide more valuable information for NER, and thus our model achieves promising performance. Third, the XLNET-based model obtains better results than the BERT-based model, which indicates that XLNET can generate more effective representations on the NER task. The reason behind this might be that XLNET combines the permutation operation with the autoregressive technology to further improve representation learning, so that XLNET can provide a better text understanding than BERT.

### Biomedical NER

We also compare our model with state-of-the-art models in the biomedical NER on the aforementioned datasets with all results reported in Table 3. There are several observations. First, we can see that our model outperforms existing methods, regardless of whether they introduce external knowledge, which further confirms the validity of our innovation in combining local and global features to enhance feature extraction. Second, although some models utilize higher-level features, e.g., BIOKMRER leverages POS labels, syntactic constituents, dependency relations, and MTM-CW employs multi-task learning to train the model, our model can achieve better results through a simple Hero-Gang structure. This means that local features extracted from the Gang module under the guidance of global information are also effective in assisting biomedical text representations and even show more significant potential than those special designs for the medical domain (i.e., domain-related multi-task learning). Third, the models using the multi-window attention (i.e., DOT-Attention and MLP-Attention) outperform those using concatenation or summation. This observation suggests that multi-window attention can elaborately weigh local features from different sliding windows to enhance feature combinations.

### Analyses

Effect of position informationRecurrent structures are able to extract both context and position information by its token-by-token manner while other network structures, including CNN and MLP, fail to encode the relative position information. Thus, to explore the effect of position information, we compare models with different structures to construct the Gang module and report the improvements of F-1 score based on different Gang modules in Figure 2. First, we can observe that models with Gang module are better than Base (i.e., BERT), where all the values in Figure 2 are positive, further illustrating the effectiveness of our innovation in combining both global and local features, no matter what type of structure is used to construct the Gang module. Second, models with LSTM and GRU outperform those with CNN and MLP, indi 

cating that recurrent structures are more promising in short sequence feature extraction. Since the recurrent structures can effectively capture position information by its token-by-token manner and help the model understand word-word relations based on their relative positions, we may conclude that position information is vital for improving performance. Third, the comparison between CNN and MLP shows the power of CNN in extracting features from sub-sequences since CNN can leverage more fine-grained features, such as n-gram.

Ablation studiesIn this subsection, we compare our multi-window model with single-window models, and the improvements compared with Base model are shown in Figure 3. We have following observations. First of all, illustrated by the comparisons among Base (i.e., BERT) and others, models with sliding windows achieve better performance, where all the improvement values in Figure 3 are positive. This illustrates that both single window and multi-window recurrent structures can help to enhance token representation and bring different degrees of improvement, which further shows the importance of local features in this task. Second, we can observe that the optimal single window sizes for different datasets are also different. For example, the optimal single window size of W17 is 5, while that for BC2GM is 7, which indicates that the best length of the local sequence depends on the characteristics of datasets to some extent. Third, compared with those models using a single window, the multi-window recurrent module obtains better performance, illustrating that features extracted from multiple sub-sequences are more effective than those captured from a single one. The reason could be that multi-window can help the model pay attention to different local context sub-sequences and give them appropriate weights through the multi-window attention mechanism, such that it can provide more reasonable local information and alleviate the impact of the characteristics of the datasets themselves.

Case StudyTo further show the validity of our model, we perform qualitative analysis on some cases with their real labels and predicted labels from different models. Figure 4 shows two cases from ON5e and BC5-C, respectively. We can observe that our model can predict more complete entities than Base. Specifically, in the first case, our model can recognize all the words in the entity _"a period of years"_ while Base model only recognizes the word _"years"_. In the second case, our model is able to identify _"Monosodium glutamate"_, but Base model regards these words as two different entities. In addition, in the first example, compared with real labels, our model can label two _"of"_ correctly with the help of local features, which are O and I-date, respectively, while Base classifies both _"of"_ as O. The sub-sequence (i.e.,_"a period of years,"_) from the second _"of"_ is usually used to describe time such that this information is able to assist the model in marking the _"of"_ as I-date. However, for the first _"of"_, its sub-sequence _"divest themselves of such speculative"_ does not contain any meaning related to the entity themes, and thus the model marks the corresponding _"of"_ as O.

Figure 3: The improvement values (%) of models with single windows or multi-window compared to Base models (BERT or BioBERT w.r.t. datasets), where 3, 5, 7, 9, 11 represents the single window size when models only use a single window to construct the Gang module.

Figure 2: The improvement values (%) compared to Base models (i.e., BERT for general domain datasets and BioBERT for biomedical datasets) in terms of F-1 score from different Gang modules, MLP, CNN, RNN, GRU, and LSTM, respectively.

 



## 5 Related Work

NER is a fundamental task in NLP Huang et al. (2015), which has drawn substantial attention over the past years and there have been many studies to address this task. Recently, deep learning has played a dominant role in NER due to its effectiveness in capturing contextual information from sequences. The recurrent neural networks (RNN), including its variants such as LSTM Hochreiter et al. (1997), and GRU Cho et al. (2014), is a promising structure for solving this task since it can effectively learn sequence information with its recurrent mechanism Ma and Hovy (2016); Huang et al. (2015); Chiu and Nichols (2016); Zhu and Wang (2019). However, it is ineffective for RNN to learn long sequences due to the gradients exploding and vanishing. Thus, Transformer-based models, such as BERT Devlin et al. (2018), BioBERT Lee et al. (2020), and XLNET Yang et al. (2019), are proposed to alleviate these problems with the help of the self-attention mechanism. Compared to RNN, Transformer is able to capture long-distance information through multiple multi-head attention layers and has achieved impressive performance in this task Nie et al. (2020); Luo et al. (2020); Yamada et al. (2020); Gui et al. (2019).

However, multi-head attention usually treats every position identically, which lead to the loss of position information. To mitigate this problem, several approaches have been proposed to advance the Transformer Dai et al. (2019); Shaw et al. (2018); Yan et al. (2019). Shaw et al. (2018) proposed cross-lingual position representation to help self-attention alleviate word order divergences in different languages and learn position information. Yan et al. (2019) introduced the directional relative positional encoding and an adapted Transformer Encoder to model the character-level and word-level features. Although these position embeddings are able to help the model learn position information, they are still not enough to solve the issue appropriately Wang et al. (2019); Huang et al. (2020); Qu et al. (2021). Besides, Transformer-based approaches cannot effectively extract local features that are also important for sequence learning tasks, and some studies have been proposed to alleviate this problem Xu et al. (2017); Li et al. (2019); Yang et al. (2019). Xu et al. (2017) proposed to use the fixed-size ordinally forgetting encoding to model sentence fragments, which is then used to predict the label for each text fragment. Li et al. (2019) utilized convolutional self-attention by producing queries and keys with causal convolution to incorporate local contextual information into the attention mechanism. To address these issues, we offer an alternative solution, namely Hero-Gang Neural model, to enhance local and position information extraction via multiple recurrent structures under the guidance of global information.

## 6 Conclusion

In this paper, we propose a novel Hero-Gang Neural (HGN) structure to effectively combine global and local features for enhancing NER. In detail, the Hero module aims to capture global understanding by a Transformer-based encoder, which is then used to guide the Gang to extract local features and relative position information through a multi-window recurrent module. Afterward, we utilize the multi-window attention to elaborately combine

Figure 4: Examples of two predicted labels from Base and Ours as well as their corresponding source sentence and real label. Note that the Base for these two cases are Bert and BioBERT, respectively.



the global information and local features for enhancing representations that are then used to predict the entity label for each token. Empirically, our proposed model achieves new state-of-the-art results on several NER benchmark datasets, including both general and biomedical domains. Besides, we compare different structures to construct the Gang model and investigate the effect of the number of sliding windows, which further illustrates the effectiveness of our proposed model.

## Acknowledgements

This work is supported by Chinese Key-Area Research and Development Program of Guangdong Province (2020B0101350001), NSFC under the project "The Essential Algorithms and Technologies for Standardized Analytics of Clinical Texts" (12026610) and the Guangdong Provincial Key Laboratory of Big Data Computing, The Chinese University of Hong Kong, Shenzhen.

## References

* I. Beltagy, K. Lo, and A. Cohan (2019)Scibert: a pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3606-3611. Cited by: SS4.
* A. Bosselut, H. Rashkin, M. Sap, C. Malaviya, A. Celikyilmaz, and Y. Choi (2019)Compet: commonsense transformers for automatic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4762-4779. Cited by: SS1.
* J. P. Chiu and E. Nichols (2016)Named entity recognition with bidirectional LSTM-cnn. Transactions of the Association for Computational Linguistics4, pp. 357-370. External Links: Document, ISSN 1878-0226, Link Cited by: SS1.
* K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio (2014)Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1724-1734. Cited by: SS1.
* Z. Dai, Z. Yang, Y. Yang, J. G. Carbonell, Q. Le, and R. Salakhutdinov (2019)Transformer-xl: attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2978-2988. Cited by: SS1.
* L. Derczynski, E. Nichols, M. van Erp, and N. Limspoatham (2017)Results of the WNUT2017 Shared Task on Novel and Emerging Entity Recognition. In Proceedings of the 3rd Workshop on Noisy User-generated Text, Vol. , pp. 140-147. Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2018)Bert: pre-training of deep bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805. Cited by: SS1.
* T. Gui, Y. Zou, Q. Zhang, M. Peng, J. Fu, Z. Wei, and X. Huang (2019)A lexicon-based graph neural network for chinese ncr. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 1040-1050. Cited by: SS1.
* B. He, Y. Guan, and R. Dai (2019)Classifying medical relations in clinical text via convolutional neural networks. Artificial intelligence in medicine93, pp. 43-49. Cited by: SS1.
* S. Hochreiter, J. Schmidhuber, and C. Elvezia (1997)Long short-term memory. Neural Computation9 (8), pp. 1735-1780. Cited by: SS1.
* X. Huang, Z. Deng, D. Li, and X. Yuan (2021)MISSFormer: an effective medical image segmentation transformer. arXiv preprint arXiv:2109.07162. Cited by: SS1.
* Z. Huang, D. Liang, P. Xu, and B. Xiang (2020)Improve Transformer Models with better relative position embeddings. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pp. 3327-3335. Cited by: SS1.
* Z. Huang, W. Xu, and K. Yu (2015) Bidirectional LSTM-CRF Models for Sequence Tagging. arXiv preprint arXiv:1508.01991. Cited by: SS1.
* D. P. Kingma and J. Ba (2014)Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: SS1.
* G. Lample, M. Ballesteros, S. Subramanian, K. Kawakami, and C. Dyer (2016)Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 260-270. Cited by: SS1.
* J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. So, and J. Kang (2020)BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics (Oxford, England)36 (4), pp. 1234. Cited by: SS1.
* S. Li, X. Jin, Y. Xuan, X. Zhou, W. Chen, Y. Wang, and X. Yan (2019)Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in Neural Information Processing Systems32, pp. 5243-5253.

 

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A Robustly Optimized BERT Pretraining Approach. _arXiv preprint arXiv:1907.11692_.
* Luo et al. (2020) Ying Luo, Fengshun Xiao, and Hai Zhao. 2020. Hierarchical Contextualized Representation for Named Entity Recognition. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 34, pages 8441-8448.
* Ma and Hovy (2016) Xuezhe Ma and Eduard Hovy. 2016. End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1064-1074.
* Morval et al. (2012) Sudha Morval, Nusrat Jahan, and Deepti Chopra. 2012. Named Entity Recognition Using Hidden Markov Model (hmm). _International Journal on Natural Language Computing (IJNLC) Vol_, 1.
* Mozharova and Loukachevitch (2016) Valerie A Mozharova and Natalia V Loukachevitch. 2016. Combining Knowledge and CRF-based Approach to Named Entity Recognition in Russian. In _International Conference on Analysis of Images, Social Networks and Texts_, pages 185-195. Springer.
* Nie et al. (2020a) Yuyang Nie, Yuanhe Tian, Yan Song, Xiang Ao, and Xiang Wan. 2020a. Improving Named Entity Recognition with Attentive Ensemble of Syntactic Information. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_, pages 4231-4245.
* Nie et al. (2020b) Yuyang Nie, Yuanhe Tian, Xiang Wan, Yan Song, and Bo Dai. 2020b. Named Entity Recognition for Social Media Texts with Semantic Augmentation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1383-1391.
* Peng et al. (2019) Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019. Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets. In _Proceedings of the 18th BioNLP Workshop and Shared Task_, pages 58-65.
* Pergola et al. (2021) Gabriele Pergola, Elena Kochkine, Lin Gui, Maria Liakata, and Yulan He. 2021. Boosting Low-Resource Biomedical QA via Entity-Aware Masking Strategies. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 1977-1985.
* Pradhan et al. (2013) Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Huwee Tou Ng, Anders Bjorkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards Robust Linguistic Analysis Using Ontonotes. In _Proceedings of the Seventeenth Conference on Computational Natural Language Learning_, pages 143-152.
* Qu et al. (2021) Anlin Qu, Jianwei Niu, and Shasha Mo. 2021. Explore Better Relative Position Embeddings from Encoding Perspective for Transformer Models. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 2989-2997.
* Sachan et al. (2018) Devendra Singh Sachan, Pengtao Xie, Minmaya Sachan, and Eric P Xing. 2018. Effective Use of Bidirectional Language Modeling for Transfer Learning in Biomedical Named Entity Recognition. In _Machine learning for healthcare conference_, pages 383-402. PMLR.
* Shaw et al. (2018) Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with Relative Position Representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_, pages 464-468.
* Strauss et al. (2016) Benjamin Strauss, Bethany Toma, Alan Ritter, Marie-Catherine De Marneffe, and Wei Xu. 2016. Results of the WNUT16 Named Entity Recognition Shared Task. In _Proceedings of the 2nd Workshop on Noisy User-generated Text (WNUT)_, pages 138-144.
* Tong et al. (2021) Y. Tong, Y. Chen, and X. Shi. 2021. A Multi-Task Approach for Improving Biomedical Named Entity recognition by incorporating multi-granularity information. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In _Proceedings of the 31st International Conference on Neural Information Processing Systems_, pages 6000-6010.
* Wang et al. (2020) Jiuniu Wang, Wenjia Xu, Xingyu Fu, Guangluan Xu, and Yirong Wu. 2020. ASTRAL: Adversarial Trained LSTM-CNN for Named Entity Recognition. _Knowledge-Based Systems_, 197:105842.
* Wang et al. (2021) Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. 2021. Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning. _arXiv preprint arXiv:2105.03654_.
* Wang et al. (2019a) Xuan Wang, Yu Zhang, Xiang Ren, Yuhao Zhang, Marinka Zitnik, Jingbo Shang, Curtis Langlotz, and Jiawei Han. 2019a. Cross-type Biomedical Named Entity Recognition with Deep Multi-task Learning. _Bioinformatics_, 35(10):1745-1752.
* Wang et al. (2019b) Zhiwei Wang, Yao Ma, Zitao Liu, and Jiliang Tang. 2019b. R-transformer: Recurrent Neural Network Enhanced Transformer. _arXiv preprint arXiv:1907.05572_.
* Wolf et al. (2020) Thomas Wolf, Julien Chaumond, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pierric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. 2020. Transformers: State-of-the-art Natural Language Processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45 

Lu Xu, Zhanming Jie, Wei Lu, and Lidong Bing. 2021. Better Feature Integration for Named Entity Recognition. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3457-3469.
* Xu et al. (2017) Mingbin Xu, Hui Jiang, and Sedtawut Watcharawitayakul. 2017. A Local Detection Approach for Named Entity Recognition and Mention Detection. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1237-1247.
* Yamada et al. (2020) Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: Deep Contextualized Entity Representations with Entity-Aware Self-Attention. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6442-6454.
* Yan et al. (2019) Hang Yan, Bocao Deng, Xiaonan Li, and Xipeng Qiu. 2019. Tener: Adapting Transformer Encoder for Named Entity Recognition. _arXiv preprint arXiv:1911.04474_.
* Yan et al. (2021) Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang, and Xipeng Qiu. 2021. A unified generative framework for various ner subtasks. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 5808-5822.
* Yang et al. (2019a) Baosong Yang, Longyue Wang, Derek F Wong, Lidia S Chao, and Zhaopeng Tu. 2019a. Convolutional Self-Attention Networks. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4040-4045.
* Yang et al. (2019b) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019b. XLnet: Generalized Autoregressive Pretraining for Language Understanding. _Advances in Neural Information Processing Systems_, 32:5753-5763.
* Yuan et al. (2021) Zheng Yuan, Yijia Liu, Chuanqi Tan, Songfang Huang, and Fei Huang. 2021. Improving Biomedical Pre-trained Language Models with Knowledge. In _Proceedings of the 20th Workshop on Biomedical Language Processing_, pages 180-190.
* Zhu and Wang (2019) Yuying Zhu and Guoxin Wang. 2019. Can-ner: Convolutional attention network for chinese named entity recognition. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 3384-3393.

 

Appendix

### Hyper-parameter Settings

We have tested several combinations of hyper-parameters in tuning our models for all NLP and Biomedical benchmark datasets (i.e., W16, W17, ON5e, BC5CDR-disease, BC2GM, and BC5CDRChem). Table 4 reports the combinations that achieve the highest F-1 score for each dataset.

 



\begin{table}
\begin{tabular}{c|c|c|c|c|c|c|c} \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Hyperr.**} & \multicolumn{4}{c|}{**NLP Data**} & \multicolumn{3}{c}{**Biomedicine Data**} \\ \cline{5-8}  & & W16 & W17 & ON5e & BC2GM & BC5-D & BC5-C \\ \hline \multirow{3}{*}{HGN (MLP)} & Window Size & \{1,3,5,7\} & \{3,5,7\} & \{5,7,9\} & \{1,3,5\} & \{5,7,11\} & \{5,7,11\} \\  & Learning Rate & 3e-5 & 5e-5 & 1e-5 & 1e-5 & 9e-6 & 1e-5 \\  & Batch Size & 32 & 32 & 32 & 32 & 32 & 32 \\ \hline \multirow{3}{*}{HGM (DOT)} & Window Size & \{3,5,7\} & \{5,7,9\} & \{3,5,7\} & \{3,5,7\} & \{5,7,9\} & \{5,7,11\} \\  & Learning Rate & 3e-5 & 5e-5 & 1e-5 & 1e-5 & 9e-6 & 9e-6 \\ \cline{1-1}  & Batch Size & 32 & 32 & 32 & 32 & 32 & 32 \\ \hline \end{tabular}
\end{table}
Table 4: The hyper-parameters for best models that we have experimented on the given datasets.