SlowGuess/NAACL_2022_OCR huggingface dataset:
When DAPT is introduced to Ours and MULTI-CELL LSTM, Ours+DAPT and MULTI-CELL LSTM+DAPT further improve the performance (with  $3.71\%$  and  $2.16\%$  improvements on averaged F1-score on CrossNER dataset), which illustrates that DAPT can narrow the gap between the
source and target domains. Since domain-related corpus contains abundant domain-specialized background information, it can help the model better understand the text in the target domain. Besides, both Ours and Ours+DAPT outperform MULTICELL LSTM+DAPT, regardless of DAPT, further demonstrating the potential of our proposed model in cross-domain NER.
4.2 Effect of Label-Aware Information
The main results are shown in Table 3. First, models incorporating label information outperform those ignoring such information (i.e., BERT and DAPT w/o LB+LC), which further confirms the validity of label information in this task. We can attribute that such information can provide valuable label-related knowledge to enhance the entity prediction. Second, on these datasets, the performance gains from our full model (i.e., BERT with LB+LC) over BERT-TAGGER on the CrossNER are larger than that of Movie and Restaurant. This observation owes to the fact that Movie and Restaurant do not share the same entity types with Conll2003, leading to a larger gap between label information from the source domain and target domain, which makes it more difficult for label features transfer. Third, our proposed framework shows its effectiveness when compared with those models that introduce the CRF layer. The reason behind this might be that our model can learn better label-related information from the source domain (including token-label and label-label relationships) and transfer it to the target domain, especially for two domains that share the same NE labels, while CRF can only recognize correlations between tags in the neighborhoods.
Moreover, we also conduct ablation studies: (1) without label context information (i.e., w/o LC), (2) without label background information (i.e., w/o LB), (3) without label context and background in
Figure 2: F1 scores of fine-grained comparisons on AI datasets. Results are averaged over three runs. The last four entity types are shared between the source and target domains.
formation (i.e., w/o LB+LC). The comparison between the base model (i.e., w/o LB+LC) and ones with LB or LC shows the effectiveness of each component in promoting cross-domain entity identification. Besides, it is observed that our full models (i.e., LB+LC) outperform w/o LB and w/o LC, which illustrates that combining label background and context information can further enhance information transfer and bring more improvements. The main reason might be that these two vectors are weighted sum over token features and label representation, respectively, with different focuses. Therefore, their combination can generate a better understanding of the label knowledge.
5.1 In-domain Performance
To test the in-domain performance of our model, we utilize the single domain dataset to train our model and evaluate it on the corresponding test set, with results reported in Table 4. We can observe that our innovation in terms of incorporating label information is also productive for the in-domain NER task, where our model achieves better performance than the corresponding baselines. It can be attributed that our model can grasp a more comprehensive understanding between text and their labels and thus boost in-domain NER. However, the improvement gains from our model over baselines on the in-domain NER task are not as significant as that on cross-domain NER. An explanation for this observation may be that, in cross-domain NER, our model can better comprehend shared NE types. Therefore, it could help the model recognize differences and find more reasonable similarities between different domains, while this advantage
MODEL	POLITICS	SCIENCE	MUSIC	LITERATURE	AI
BERT†	66.56	63.73	66.59	59.95	50.37
OURS	68.13	66.21	68.75	61.37	53.09
DAPT†	70.45	67.59	73.39	64.96	56.36
OURS+DAPT	71.83	69.23	74.79	66.35	58.12
Table 4: F1 scores with respect to in-domain NER.
may not be helpful for in-domain NER.
5.2 Fine-grained Comparison
We further investigate the fine-grained comparison on the AI dataset and visualize the results in Figure 2. We can see that our model obtains better performance in most entity types, regardless of whether DAPT is used, which indicates that the improvements gained from label-aware information are consistent across various entity classes. All shared entity types (i.e., the last four entity types) achieve increased performance by our models, indicating that our model can grasp more useful label information from the source domain and effectively transfer them to the target domain. However, for all non-shared entity types, our model leads to a slight decrease on a few entity types (e.g., COUNTRY). We find that COUNTRY is similar with LOCATION and Conll2003 annotates some countries as the LOCATION while CrossNER tends to label them as the COUNTRY. For example, "Netherlands" in Conll2003 is the LOCATION, whereas, in the AI dataset, it is marked as the COUNTRY. Hence, the label information learned from the source domain may contribute to mis-classification about COUNTRY in the target domain. It can also explain performance drop in entity category RESEARCHER since it is easily confused with PERSON.
Figure 3: The F1 scores from Ours and Ours+DAPT against the label embedding dimension, where the model is tested on the AI dataset.
5.3 Effect of Label Embedding Dimension
To demonstrate the impacts of label embedding size, we train our model by varying the embedding dimension  $d_{2}$  from 10 to 1000, as shown in Figure 3. It is observed that increasing the embedding size performs a better performance when the dimension is relatively small (i.e.,  $d_{2} \leq 100$  for Ours+DAPT and  $d_{2} \leq 150$  for Ours). It indicates that, within this range, larger embedding can bring more valuable label information. However, when the dimension becomes too large, the performance gradually drops. It can be explained that a too large embedding matrix is difficult to be trained, resulting in redundant noise and degraded model performance. In addition, our model only introduces relatively small parameters when incorporating label-aware information. Especially when our models obtain the best results at  $d_{2} = 100$  and 150, their introduced extra parameters are  $0.88\%$  and  $1.41\%$  compared to the base model.
5.4 Effect of Data Size
To explore the impact of the target domain data size, we conduct experiments on different amounts of target training data (i.e., increasing from 10 to 100 samples) based on best-performing settings. The results are shown in Figure 4. With the data size increasing, all models gradually obtain better F1 scores, which illustrates that data scale plays an important role in the NER task. Besides, it is observed that both Ours and Ours+DAPT outperform corresponding baselines (i.e., BERT and DAPT) no matter how many samples we select, which further confirms the effectiveness of incorporating label information into cross-domain NER.
Figure 4: F1 scores of BERT, Ours, DAPT, and Ours+DAPT with different amounts of AI data in the target domain.
In NLP, NER aims to identify entities from unstructured text, which has been studied widely over the past decades. Recently, neural networks have significantly improved the performance of NER, owing to their strong ability in feature extraction (Huang et al., 2015; Lample et al., 2016; Ma and Hovy, 2016; Yan et al., 2019; Devlin et al., 2019; Luo et al., 2020; Wang et al., 2021; Yamada et al., 2020; Yan et al., 2021). For example, Huang et al. (2015); Lample et al. (2016) combined Bi-LSTM with a CRF layer to enhance NER. Devlin et al. (2019); Yamada et al. (2020); Yan et al. (2019) further introduced the Transformer-based (Vaswani et al., 2017) encoders to extract more effective information from the sequence, which is then used to facilitate NER. However, although these models achieved great performance, they required large-scale labeled training data to adapt to different domains. Therefore, cross-domain NER has drawn substantial attention in recent years and gradually become one of the hot research topics in NLP. Many approaches have been proposed to enhance cross-domain NER (Pan et al., 2013; Jia et al., 2019; Jia and Zhang, 2020; Liu et al., 2020b; Chen and Moschitti, 2019). For example, Jia et al. (2019) utilized a parameter generation network to perform cross-domain and cross-task knowledge transfer and employed multi-task learning to combine NER and language modeling tasks. Furthermore, Jia and Zhang (2020) presented a multi-cell compositional LSTM structure that incorporated the entity type by a separate cell state to enhance the cross-domain NER. Compared with these studies, our model provides a simple but effective solution for addressing
cross-domain NER by improving label information transfer and predicting current labels through corresponding tokens and previous labels together.
In this paper, we have proposed a novel framework for cross-domain NER to enhance the relationship between the source text and labels and improve label information transfer, where each NE label is jointly predicted by corresponding token and previous NE labels. We not only adopt a commonly-used pre-trained model to extract token representation, but also introduce a random initialized embedding matrix and Bi-LSTM-based label encoder to model the label sequence generated from previous steps. After that, we construct two different attention between hidden states of Bi-LSTM and token representations to produce label background and context information, which are then concatenated as label-aware information and applied to predict labels. Thanks to this design, the label information can be effectively transferred from the source to the target domain. Comprehensive experimental results on several benchmark datasets illustrate the effectiveness of our model, which achieves significant improvements over existing methods.
This work is supported by Chinese Key-Area Research and Development Program of Guangdong Province (2020B0101350001), NSFC under the project "The Essential Algorithms and Technologies for Standardized Analytics of Clinical Texts" (12026610) and the Guangdong Provincial Key Laboratory of Big Data Computing, The Chinese University of Hong Kong, Shenzhen.
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual String Embeddings for Sequence Labeling. In Proceedings of the 27th international conference on computational linguistics, pages 1638-1649.
Lingzhen Chen and Alessandro Moschitti. 2019. Transfer Learning for Sequence Labeling Using Source Model and Target Data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6260-6267.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the
North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don't Stop Pretraining: Adapt Language Models to Domains and Tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360.
Sepp Hochreiter, J urgen Schmidhuber, and Corso Elvezia. 1997. Long Short-term Memory. Neural Computation, 9(8):1735-1780.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF Models for Sequence Tagging. arXiv preprint arXiv:1508.01991.
Chen Jia, Xiaobo Liang, and Yue Zhang. 2019. Cross-domain NER using Cross-Domain Language Modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2464-2474.
Chen Jia and Yue Zhang. 2020. Multi-Cell Compositional LSTM for NER Domain Adaptation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5906-5917.
Diederik P Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR (Poster).
John D Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. In Proceedings of the Eighteenth International Conference on Machine Learning, pages 282-289.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural Architectures for Named Entity Recognition. In Proceedings of NAACL-HLT, pages 260-270.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880.
Bill Yuchen Lin and Wei Lu. 2018. Neural Adaptation Layers for Cross-domain Named Entity Recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2012-2022.
Jingjing Liu, Panupong Pasupat, Scott Cyphers, and Jim Glass. 2013a. Asgard: A portable architecture for multilingual dialogue systems. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 8386-8390. IEEE.
Jingjing Liu, Panupong Pasupat, Yining Wang, Scott Cyphers, and Jim Glass. 2013b. Query understanding enhanced by hierarchical parsing structures. In 2013 IEEE Workshop on Automatic Speech Recognition and Understanding, pages 72-77. IEEE.
Zihan Liu, Genta Indra Winata, Peng Xu, and Pascale Fung. 2020a. Coach: A Coarse-to-Fine Approach for Cross-Domain Slot Filling. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 19-25.
Zihan Liu, Yan Xu, Tiezheng Yu, Wenliang Dai, Zwei Ji, Samuel Cahyawijaya, Andrea Madotto, and Pascale Fung. 2020b. CrossNER: Evaluating Cross-Domain Named Entity Recognition.
Ying Luo, Fengshun Xiao, and Hai Zhao. 2020. Hierarchical Contextualized Representation for Named Entity Recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8441-8448.
Xuezhe Ma and Eduard Hovy. 2016. End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1064-1074.
Sinno Jialin Pan, Zhiqiang Toh, and Jian Su. 2013. Transfer Joint Embedding for Cross-domain Named Entity Recognition. ACM Transactions on Information Systems (TOIS), 31(2):1-27.
Erik Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the Conll-2003 Shared Task: Language-Independent Named Entity Recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142-147.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in neural information processing systems, pages 5998-6008.
Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. 2021. Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning. arXiv preprint arXiv:2105.03654.
Yequan Wang, Minlie Huang, Xiaoyan Zhu, and Li Zhao. 2016. Attention-based LSTM for aspect-level sentiment classification. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 606-615.
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6442-6454.
Hang Yan, Bocao Deng, Xiaonan Li, and Xipeng Qiu. 2019. TENER: Adapting Transformer Encoder for Named Entity Recognition. arXiv preprint arXiv:1911.04474.
Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang, and Xipeng Qiu. 2021. A Unified Generative Framework for Various NER Subtasks. arXiv preprint arXiv:2106.01223.
MODEL	HY.	DATA
CROSSNER	MOVIE	RESTAURANT
OURS	BS	16	16	16
LR	5e-5	5e-5	1e-5
ME	100	100	100
Table 5: The best hyper-parameters that we used in our experiments. Bs, LR, and ME represent the batch size, learning rate, and max epochs, respectively.
A.1 Hyper-parameter Settings
We have tested several combinations of hyperparameters in tuning our models on CrossNER, Movie and Restaurant. Table 5 reports the combinations that achieve the highest F-1 score for each dataset.