SlowGuess/NAACL_2022_OCR huggingface dataset:
dependency relations, and MTM-CW employs multi-task learning to train the model, our model can achieve better results through a simple HeroGang structure. This means that local features extracted from the Gang module under the guidance of global information are also effective in assisting biomedical text representations and even show more significant potential than those special designs for the medical domain (i.e., domain-related multi-task learning). Third, the models using the multi-window attention (i.e., DOT-Attention and MLP-Attention) outperform those using concatenation or summation. This observation suggests that multi-window attention can elaborately weigh local features from different sliding windows to enhance feature combinations.
Effect of position information Recurrent structures are able to extract both context and position information by its token-by-token manner while other network structures, including CNN and MLP, fail to encode the relative position information. Thus, to explore the effect of position information, we compare models with different structures to construct the Gang module and report the improvements of F-1 score based on different Gang modules in Figure 2. First, we can observe that models with Gang module are better than Base (i.e., BERT), where all the values in Figure 2 are positive, further illustrating the effectiveness of our innovation in combining both global and local features, no matter what type of structure is used to construct the Gang module. Second, models with LSTM and GRU outperform those with CNN and MLP, indi
Figure 2: The improvement values  $(\%)$  compared to Base models (i.e., BERT for general domain datasets and BioBERT for biomedical datasets) in terms of F-1 score from different Gang modules, MLP, CNN, RNN, GRU, and LSTM, respectively.
cating that recurrent structures are more promising in short sequence feature extraction. Since the recurrent structures can effectively capture position information by its token-by-token manner and help the model understand word-word relations based on their relative positions, we may conclude that position information is vital for improving performance. Third, the comparison between CNN and MLP shows the power of CNN in extracting features from sub-sequences since CNN can leverage more fine-grained features, such as n-gram.
In this subsection, we compare our multi-window model with single-window models, and the improvements compared with Base model are shown in Figure 3. We have following observations. First of all, illustrated by the comparisons among Base (i.e., BERT) and others, models with sliding windows achieve better performance, where all the improvement values in Figure 3 are positive. This illustrates that both single window and multi-window recurrent structures can help to enhance token representation and bring different degrees of improvement, which further shows the importance of local features in this task. Second, we can observe that the optimal single window sizes for different datasets are also different. For example, the optimal single window size of W17 is 5, while that for BC2GM is 7, which indicates that the best length of the local sequence depends on the characteristics of datasets to some extent. Third, compared with those models using a single window, the multi-window recurrent module obtains better performance, illustrating that features
Figure 3: The improvement values  $(\%)$  of models with single windows or multi-window compared to Base models (BERT or BioBERT w.r.t. datasets), where 3, 5, 7, 9, 11 represents the single window size when models only use a single window to construct the Gang module.
extracted from multiple sub-sequences are more effective than those captured from a single one. The reason could be that multi-window can help the model pay attention to different local context sub-sequences and give them appropriate weights through the multi-window attention mechanism, such that it can provide more reasonable local information and alleviate the impact of the characteristics of the datasets themselves.
Case Study To further show the validity of our model, we perform qualitative analysis on some cases with their real labels and predicted labels from different models. Figure 4 shows two cases from ON5e and BC5-C, respectively. We can observe that our model can predict more complete entities than Base. Specifically, in the first case, our model can recognize all the words in the entity "a period of years" while Base model only recognizes the word "years". In the second case, our model is able to identify "Monosodium glutamate", but Base model regards these words as two different entities. In addition, in the first example, compared with real labels, our model can label two "of" correctly with the help of local features, which are O and I-date, respectively, while Base classifies both "of" as O. The sub-sequence (i.e., "a period of years,") from the second "of" is usually used to describe time such that this information is able to assist the model in marking the "of" as I-date. However, for the first "of", its sub-sequence "divest themselves of such speculative" does not contain any meaning related to the entity themes, and thus the model marks the corresponding "of" as O.
Figure 4: Examples of two predicted labels from BASE and Ours as well as their corresponding source sentence and real label. Note that the BASE for these two cases are BERT and BIOBERT, respectively.
NER is a fundamental task in NLP (Huang et al., 2015), which has drawn substantial attention over the past years and there have been many studies to address this task. Recently, deep learning has played a dominant role in NER due to its effectiveness in capturing contextual information from sequences. The recurrent neural networks (RNN), including its variants such as LSTM (Hochreiter et al., 1997), and GRU (Cho et al., 2014), is a promising structure for solving this task since it can effectively learn sequence information with its recurrent mechanism (Ma and Hovy, 2016; Huang et al., 2015; Chiu and Nichols, 2016; Zhu and Wang, 2019). However, it is ineffective for RNN to learn long sequences due to the gradients exploding and vanishing. Thus, Transformer-based models, such as BERT (Devlin et al., 2018), BioBERT (Lee et al., 2020), and XLNET (Yang et al., 2019b), are proposed to alleviate these problems with the help of the self-attention mechanism. Compared to RNN, Transformer is able to capture long-distance information through multiple multi-head attention layers and has achieved impressive performance in this task (Nie et al., 2020b; Luo et al., 2020; Yamada et al., 2020; Gui et al., 2019).
However, multi-head attention usually treats every position identically, which lead to the loss of position information. To mitigate this problem, several approaches have been proposed to advance the Transformer (Dai et al., 2019; Shaw et al., 2018; Yan et al., 2019). Shaw et al. (2018) proposed cross-lingual position representation to help self-attention alleviate word order divergences in different languages and learn position information. Yan
et al. (2019) introduced the directional relative positional encoding and an adapted Transformer Encoder to model the character-level and word-level features. Although these position embeddings are able to help the model learn position information, they are still not enough to solve the issue appropriately (Wang et al., 2019b; Huang et al., 2020; Qu et al., 2021). Besides, Transformer-based approaches cannot effectively extract local features that are also important for sequence learning tasks, and some studies have been proposed to alleviate this problem (Xu et al., 2017; Li et al., 2019; Yang et al., 2019a). Xu et al. (2017) proposed to use the fixed-size ordinally forgetting encoding to model sentence fragments, which is then used to predict the label for each text fragment. Li et al. (2019) utilized convolutional self-attention by producing queries and keys with causal convolution to incorporate local contextual information into the attention mechanism. To address these issues, we offer an alternative solution, namely Hero-Gang Neural model, to enhance local and position information extraction via multiple recurrent structures under the guidance of global information.
In this paper, we propose a novel Hero-Gang Neural (HGN) structure to effectively combine global and local features for enhancing NER. In detail, the Hero module aims to capture global understanding by a Transformer-based encoder, which is then used to guide the Gang to extract local features and relative position information through a multi-window recurrent module. Afterward, we utilize the multi-window attention to elaborately combine
the global information and local features for enhancing representations that are then used to predict the entity label for each token. Empirically, our proposed model achieves new state-of-the-art results on several NER benchmark datasets, including both general and biomedical domains. Besides, we compare different structures to construct the Gang model and investigate the effect of the number of sliding windows, which further illustrates the effectiveness of our proposed model.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A Pretrained Language Model for Scientific Text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3606-3611.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. Comet: Commonsense Transformers for Automatic Knowledge Graph Construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762-4779.
Jason PC Chiu and Eric Nichols. 2016. Named Entity Recognition with Bidirectional LSTM-CNNs. Transactions of the Association for Computational Linguistics, 4:357-370.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gul- cehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder- Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724-1734.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive Language Models beyond a Fixed-length Context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988.
Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the WNUT2017 Shared Task on Novel and Emerging Entity Recognition. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 140-147.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805.
Tao Gui, Yicheng Zou, Qi Zhang, Minlong Peng, Jinlan Fu, Zhongyu Wei, and Xuan-Jing Huang. 2019. A lexicon-based graph neural network for chinese ner. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1040-1050.
Bin He, Yi Guan, and Rui Dai. 2019. Classifying Medical Relations in Clinical Text via Convolutional Neural Networks. Artificial intelligence in medicine, 93:43-49.
Sepp Hochreiter, J urgen Schmidhuber, and Corso Elvezia. 1997. Long Short-term Memory. Neural Computation, 9(8):1735-1780.
Xiaohong Huang, Zhifang Deng, Dandan Li, and Xueguang Yuan. 2021. MISSFormer: An Effective Medical Image Segmentation Transformer. arXiv preprint arXiv:2109.07162.
Zhiheng Huang, Davis Liang, Peng Xu, and Bing Xiang. 2020. Improve Transformer Models with Better Relative Position Embeddings. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 3327-3335.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF Models for Sequence Tagging. arXiv preprint arXiv:1508.01991.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural Architectures for Named Entity Recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 260-270.
J Lee, W Yoon, S Kim, D Kim, S Kim, CH So, and J Kang. 2020. Biobert: a Pre-trained Biomedical Language Representation Model for Biomedical Text Mining. Bioinformatics (Oxford, England), 36(4):1234.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. 2019. Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting. Advances in Neural Information Processing Systems, 32:5243-5253.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692.
Ying Luo, Fengshun Xiao, and Hai Zhao. 2020. Hierarchical Contextualized Representation for Named Entity Recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8441-8448.
Xuezhe Ma and Eduard Hovy. 2016. End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1064-1074.
Sudha Morwal, Nusrat Jahan, and Deepti Chopra. 2012. Named Entity Recognition Using Hidden Markov Model (hmm). International Journal on Natural Language Computing (IJNLC) Vol. 1.
Valerie A Mozharova and Natalia V Loukachevitch. 2016. Combining Knowledge and CRF-based Approach to Named Entity Recognition in Russian. In International Conference on Analysis of Images, Social Networks and Texts, pages 185-195. Springer.
Yuyang Nie, Yuanhe Tian, Yan Song, Xiang Ao, and Xiang Wan. 2020a. Improving Named Entity Recognition with Attentive Ensemble of Syntactic Information. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 4231-4245.
Yuyang Nie, Yuanhe Tian, Xiang Wan, Yan Song, and Bo Dai. 2020b. Named Entity Recognition for Social Media Texts with Semantic Augmentation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1383-1391.
Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019. Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets. In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 58-65.
Gabriele Pergola, Elena Kochkina, Lin Gui, Maria Liakata, and Yulan He. 2021. Boosting Low-Resource Biomedical QA via Entity-Aware Masking Strategies. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1977-1985.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards Robust Linguistic Analysis Using Ontonotes. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 143-152.
Anlin Qu, Jianwei Niu, and Shasha Mo. 2021. Explore Better Relative Position Embeddings from Encoding Perspective for Transformer Models. In Proceedings
of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2989-2997.
Devendra Singh Sachan, Pengtao Xie, Mrinmaya Sachan, and Eric P Xing. 2018. Effective Use of Bidirectional Language Modeling for Transfer Learning in Biomedical Named Entity Recognition. In Machine learning for healthcare conference, pages 383-402. PMLR.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with Relative Position Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 464-468.
Benjamin Strauss, Bethany Toma, Alan Ritter, Marie-Catherine De Marneffe, and Wei Xu. 2016. Results of the WNUT16 Named Entity Recognition Shared Task. In Proceedings of the 2nd Workshop on Noisy User-generated Text (WNUT), pages 138-144.
Y. Tong, Y. Chen, and X. Shi. 2021. A Multi-Task Approach for Improving Biomedical Named Entity recognition by incorporating multi-granularity information. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6000-6010.
Jiuniu Wang, Wenjia Xu, Xingyu Fu, Guanglian Xu, and Yirong Wu. 2020. ASTRAL: Adversarial Trained LSTM-CNN for Named Entity Recognition. Knowledge-Based Systems, 197:105842.
Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. 2021. Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning. arXiv preprint arXiv:2105.03654.
Xuan Wang, Yu Zhang, Xiang Ren, Yuhao Zhang, Marinka Zitnik, Jingbo Shang, Curtis Langlotz, and Jiawei Han. 2019a. Cross-type Biomedical Named Entity Recognition with Deep Multi-task Learning. Bioinformatics, 35(10):1745-1752.
Zhiwei Wang, Yao Ma, Zitao Liu, and Jiliang Tang. 2019b. R-transformer: Recurrent Neural Network Enhanced Transformer. arXiv preprint arXiv:1907.05572.
Thomas Wolf, Julien Chaumont, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pierric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. 2020. Transformers: State-of-the-art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45.
Lu Xu, Zhanming Jie, Wei Lu, and Lidong Bing. 2021. Better Feature Integration for Named Entity Recognition. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3457-3469.
Mingbin Xu, Hui Jiang, and Sedtawut Watcharawittayakul. 2017. A Local Detection Approach for Named Entity Recognition and Mention Detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1237-1247.
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: Deep Contextualized Entity Representations with Entity-Aware Self-Attention. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6442-6454.
Hang Yan, Bocao Deng, Xiaonan Li, and Xipeng Qiu. 2019. Tener: Adapting Transformer Encoder for Named Entity Recognition. arXiv preprint arXiv:1911.04474.
Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang, and Xipeng Qiu. 2021. A unified generative framework for various ner subtasks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5808-5822.
Baosong Yang, Longyue Wang, Derek F Wong, Lidia S Chao, and Zhaopeng Tu. 2019a. Convolutional Self-Attention Networks. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4040-4045.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019b. Xlnet: Generalized Autoregressive Pretraining for Language Understanding. Advances in Neural Information Processing Systems, 32:5753-5763.
Zheng Yuan, Yijia Liu, Chuanqi Tan, Songfang Huang, and Fei Huang. 2021. Improving Biomedical Pre-trained Language Models with Knowledge. In Proceedings of the 20th Workshop on Biomedical Language Processing, pages 180-190.
Yuying Zhu and Guoxin Wang. 2019. Can-ner: Convolutional attention network for chinese named entity recognition. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3384-3393.
We have tested several combinations of hyperparameters in tuning our models for all NLP and Biomedical benchmark datasets (i.e., W16, W17, ON5E, BC5CDR-disease, BC2GM, and BC5CDR-chem). Table 4 reports the combinations that achieve the highest F-1 score for each dataset.
MODEL	HYPTER.	NLP DATA	BIOMEDICINE DATA
W16	W17	ON5E	BC2GM	BC5-D	BC5-C
HGN (MLP)	Window Size	{1,3,5,7}	{3,5,7}	{5,7,9}	{1,3,5}	{5,7,11}	{5,7,11}
Learning Rate	3e-5	5e-5	1e-5	1e-5	9e-6	1e-5
Batch Size	32	32	32	32	32	32
HGM (DOT)	Window Size	{3,5,7}	{5,7,9}	{3,5,7}	{3,5,7}	{5,7,9}	{5,7,11}
Learning Rate	3e-5	5e-5	1e-5	1e-5	9e-6	9e-6
Batch Size	32	32	32	32	32	32
Table 4: The hyper-parameters for best models that we have experimented on the given datasets.