SlowGuess/NAACL_2022_OCR huggingface dataset:
Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, and Tom Kwiatkowski. 2020. Entities as experts: Sparse memory access with entity supervision. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 4937-4951.
Matthias Fey and Jan E. Lenssen. 2019. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds.
Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A. Smith, and Luke Zettlemoyer. 2021. Demix layers: Disentangling domains for modular language modeling. arXiv preprint, arXiv:2108.05036.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 8342-8360.
William L. Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 1024-1034.
Xiaochuang Han and Jacob Eisenstein. 2019. Unsupervised domain adaptation of contextualized embeddings for sequence labeling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 4237-4247.
Dan Hendrycks and Kevin Gimpel. 2016. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. arXiv preprint, arXiv:1606.08415.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pages 2790-2799.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 328-339.
Xun Huang and Serge J. Belongie. 2017. Arbitrary style transfer in real-time with adaptive instance normalization. In IEEE International Conference on
Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 1510-1519. IEEE Computer Society.
Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2020. What disease does this patient have? A large-scale open domain question answering dataset from medical exams. arXiv preprint, arXiv:2009.13081.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinform., 36(4):1234-1240.
Seanie Lee, Minki Kang, Juho Lee, and Sung Ju Hwang. 2021. Learning to perturb word embeddings for out-of-distribution QA. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 5583-5595.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint, arXiv:1907.11692.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018. Mixed precision training. In ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings.
George A. Miller. 1995. Wordnet: A lexical database for english. Commun. ACM, 38(11):39-41.
Vinod Nair and Geoffrey E. Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), June 21-24, 2010, Haifa, Israel, pages 807-814.
Anusri Pampari, Preethi Raghavan, Jennifer J. Liang, and Jian Peng. 2018. emrqa: A large corpus for question answering on electronic medical records. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 2357-2368.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z.
Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 8024-8035.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. 2018. Film: Visual reasoning with a general conditioning layer. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 3942-3951.
Matthew E. Peters, Mark Neumann, Robert L. Logan IV, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 43-54.
Yujia Qin, Yankai Lin, Ryuichi Takanobu, Zhiyuan Liu, Peng Li, Heng Ji, Minlie Huang, Maosong Sun, and Jie Zhou. 2021. ERICA: improving entity and relation understanding for pre-trained language models via contrastive learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 3350-3363.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-140:67.
Sebastian Ruder. 2019. Neural Transfer Learning for Natural Language Processing. Ph.D. thesis, National University of Ireland, Galway.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning, CoNLL 2003, Held in cooperation with HLT-NAACL 2003, Edmonton, Canada, May 31 - June 1, 2003, pages 142-147. ACL.
Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, pages 4603-4611.
Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015. End-to-end memory networks. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 2440-2448.
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. Newsqa: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, Rep4NLP@ACL 2017, Vancouver, Canada, August 3, 2017, pages 191-200.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph attention networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings.
Pat Verga, Haitian Sun, Livio Baldini Soares, and William W. Cohen. 2021. Adaptable and interpretable neural memoryover symbolic knowledge. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 3678-3691.
Denny Vrandecic and Markus Krötzsch. 2014. Wiki-data: a free collaborative knowledgebase. Commun. ACM, 57(10):78-85.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In EMNLP 2020 - Demos, Online, November 16-20, 2020, pages 38-45.
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: deep contextualized entity representations with entity-aware self-attention. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 6442-6454.
An Yang, Quan Wang, Jing Liu, Kai Liu, Yajuan Lyu, Hua Wu, Qiaoqiao She, and Sujian Li. 2019. Enhancing pre-trained language representations with
rich knowledge for machine reading comprehension. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 2346-2357.
Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. Docred: A large-scale document-level relation extraction dataset. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 764-777.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 3320-3328.
Xiang Yue, Bernal Jimenez Gutierrez, and Huan Sun. 2020. Clinical reading comprehension: A thorough analysis of the emrqa dataset. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 4474-4486.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: enhanced language representation with informative entities. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 1441-1451.
Figure 6: Visual diagram of the KG construction pipeline used in this work. The entity format is composed of its corresponding text in the data, its character-level mention boundary, and its wikidata id. The fact format is composed of the head, relation, and tail, where head and tail entities are represented with their wikidata ids following the entity format.
Hyperparameters	NewsQA	Relation	Medication	CoNLL-2003	WNUT-17	NCBI-Disease
LM for Relation Extraction	BERT-base-uncased 0.1
Threshold on Relation Extraction
Size of Entity Memory	62823	5724	4635	10288	101	3502
The location of KFM	11	11	11	8	9, 11	8, 10
Table 8: Hyperparameters for Knowledge Graph (Top) and KALA (Bottom) on six datasets we used. The reported performances on main paper are measured with the above settings.
A Details on KG Construction
In this work, we propose to use the Knowledge Graph (KG) that can define the relational information among entities that only appear in each dataset. However, unfortunately, most of the task datasets do not contain such relational facts on its context, thus we need to construct them manually to obtain the knowledge graph. In this section, we explain the way of constructing the knowledge graph that we used, consisting of facts of entities for each context in the task dataset.
Relation extraction is the way how we obtain the factual knowledge from the text of the target dataset. To do so, we first need to extract entities and their corresponding mentions from the text, and then link it to the existing entities in wikidata (Vrandecic and Krötzsch, 2014). In order to do this, we use the existing library named as spaCy<sup>5</sup>, and opensourced implementation of Entity Linker<sup>6</sup>. To sum up, in our work, a set of entities  $\mathcal{E}^{(i)}$  and corresponding mentions  $\mathcal{M}^{(i)}$  for the given input  $\boldsymbol{x}^{(i)}$  are obtained through this step. Regarding a concrete example, please see format (a) in Figure 6. In the example, "Text" indicates the entity mention within the input  $\boldsymbol{x}$ , the "start" and "end" indicates its mention position denoted as  $(m^{\alpha}, m^{\omega})$ , and "id" indicates the wikidata id for the entity identification used in the next step.
To extract the relation among entities that we obtained above, we use the scheme of Relation Extraction (RE). In other words, we use the trained
RE model to build our own knowledge base (KB) instead of using the existing KG directly from the existing general-domain KB<sup>7</sup>. Specifically, we first fine-tune the BERT-base model (Devlin et al., 2019) for 2 epochs with 600k distantly supervised data used in Qin et al. (2021), where the Wikipedia document and the Wikidata triplets are aligned. Then, we use the fine-tuned BERT model to extract the relations between entity pairs in the text. We use the model with a simple bilinear layer on top of it, which is widely used scheme in the relation extraction literature (Yao et al., 2019). For an example of the extracted fact, please see format (b) in Figure 6. In the example, “h” denotes the wikidata id of the head entity, “r” denotes the wikidata id of the extracted relation, and “t” denotes the wikidata id of the tail entity. In the relation extraction, the model returns the categorical distribution over the top 100 frequent relations. In general, the relation of top-1 probability is used as the relation for the corresponding entity pair. However, this approach sometimes results in predicting no Relation on most entity pairs. Thus, to obtain more relations, we further use the relation of top-2 probability in the case where no Relation has a top-1
<sup>7</sup>We faced several problems here. First of all, most KBs such as Wikidata are less informative, especially for the entities included in the domain-specific context (e.g., News, Medical records). It only has a few facts for each context of domain-specific tasks, although we can find a lot of entities included in the context. Second, the entity linker is imperfect. Due to the wrongly linked entity to the wikidata, even existing relations in the KG are ignored a lot. Therefore, we instead use a trained neural network to effectively extract the relations between entities, instead of direct querying to obtain facts.
$^{6}$ https://github.com/egerber/spaCy-entity-linker
Dataset	# Context	Training	Validation	Test
C. Length	# Question	# Context	C. Length	# Question	# Context	C. Length	# Question
NewsQA	11428	655.7	74160	-	-	-	106	625.8	674
Relation	296	1386.1	6162	42	1206.6	321	85	1467.7	802
Medication	182	1737.3	7518	26	1626.5	1858	53	2005.0	4005
Table 9: QA dataset statistics. We report the number of contexts and questions (i.e., # Context and # Question), with the average length of contexts (i.e., C. Length) where the length is measured as the number of tokens after wordpiece tokenization.
probability but the top-2 probability is larger than a certain threshold (e.g.,  $>0.1$ ). In Figure 6, we summarize our KG construction pipeline. In Table 8, we report the hyperparameters related to our KG construction.
In this section, we introduce the detailed setups for our models and baselines used in Table 1, 2, and 4.
We use the Pytorch (Paszke et al., 2019) for the implementation of all models. Also, to easily implement the language model, we use the huggingface library (Wolf et al., 2020) containing various transformer-based pre-trained language models (PLMs) and their checkpoints.
Details for KALA In this paragraph, we describe the implementation details of the components, such as four linear layers in the proposed KFM, architectural specifications in the attention-based GNN, and initialization of both the entity memory and relational embeddings, in the following. In terms of the functions  $h_1, h_2, h_3$ , and  $h_4$  in the KFM of Equation 2, we use two linear layers with the ReLU (Nair and Hinton, 2010) activation function, where the dimension is set to 768.
For relational retrieval, we implement the novel GNN model based on GATv2 (Brody et al., 2021) provided by the torch-geometric package (Fey and Lenssen, 2019). Specifically, we stack two GNN layers with the RELU activation function and also use the dropout with a probability of 0.1. For attention in our GNN, we mask the nodes of the null entity, so that the attention score becomes zero for them. Moreover, to obtain the context representation of the entity (See Footnote 3 in the main paper) used in the GNN attention, we use the scatter operation<sup>8</sup> for reduced computational cost.
For Entity Memory, we experimentally found that initializing the embeddings of the entity memory with the contextualized features obtained from
the pre-trained language model could be helpful. Therefore, the dimension of the entity embedding is set to the same as the language model  $d = 768$ . For relation embeddings, we randomly initialize them, where the dimension size is set to 128.
Location of KLM in the PLM Note that, the number and location of the KFM layers inside the PLM are hyperparameters. However, we empirically found that inserting one to three KFM layers at the end of the PLM (i.e., after the 9th - 11th layers of the BERT-base language model) is beneficial to the performance (See Appendix C.4 for experiments on diverse layer locations).
Here we describe the dataset details with its statistics for two different tasks: extractive question answering (QA) and named entity recognition (NER).
Question Answering We evaluate models on three domain-specific datasets: NewsQA, Relation, and Medication. Notably, NewsQA (Trischler et al., 2017) is curated from CNN news articles. Relation and Medication are originally part of the emrQA (Pampari et al., 2018), which is an automatically constructed question answering dataset based on the electrical medical record from n2c2 challenges<sup>9</sup>. However, Yue et al. (2020) extract two major subsets by dividing the entire dataset into Relation and Medication and suggest the usage of sampled questions from the original emrQA dataset. Following the suggestion of Yue et al. (2020), we use only  $1\%$  of generated questions of Relation for training, validation, and testing. Also, we only use  $1\%$  of generated questions of Medication for training and use  $5\%$  of generated questions of Medication for validation and testing. Since the original emrQA is automatically generated based on templates, the quality is poor – it means that the original emrQA dataset was inappropriate to evaluate the ability of the model to reason over the clinical text since the most of questions can be
$^{8}$ https://github.com/rustyls/pytorch_scatter
<sup>9</sup>https://portal.dbmi.hms.harvard.edu/projects/n2c2-nlp/
Hyperparameters	NewsQA	Relation	Medication	CoNLL-2003	WNUT-17	NCBI-Disease	Generative NewsQA
Fine-tuning
Language Model	BERT-base-uncased	T5-small
Maximum Sequence Length	384	384	384	128	128	128	512
Batch Size	12	12	12	32	32	32	64
Training Epochs	2	2	2	20	20	20	4
Optimizer	AdamW	Adafactor
Learning rate	3e-5	3e-5	3e-5	5e-5	5e-5	5e-5	1e-4
Weight Decay	0.01	0.01	0.01	0	0	0	-
LR decay Warmup rate	0.06	0.06	0.06	0	0	0	-
Half Precision	Yes	Yes	Yes	No	No	No	No
Task-Adaptive Pre-training (TAPT)
Maximum Sequence Length	384	384	384	128	128	128	384
Batch Size	12	12	12	32	32	32	64
Training Epochs	1	1	1	3	3	3	4
Training Epochs (RecAdam)	3	1	1	3	3	3	4
Optimizer	AdamW	Adafactor
Learning rate	5e-5	1e-3
Weight Decay	0.01	0.01	0.01	0	0	0	-
LR decay Warmup rate	0.06	0.06	0.06	0	0	0	-
Half Precision	Yes	Yes	Yes	No	No	No	No
Table 10: Hyperparameters for Fine-tuning (Top) and TAPT (Bottom) on six datasets (+ generative QA) we used for reporting the performances in the main paper. Note that the Fine-tuning setup is applied to all methods including KALA.
Dataset	Training	Validation	Test
# Context	C. Length	# Context	C. Length	# Context	C. Length
CoNLL-2003	14,041	19.95	3,250	21.36	3,453	18.77
WNUT-17	3,394	31.32	1,009	19.28	1,287	30.58
NCBI-Disease	5,433	34.36	924	35.00	941	35.50
answered by the simple text matching. To overcome this limitation, Yue et al. (2020) suggests two ways to make the task more difficult. First, they divide the question templates into easy and hard versions and then use the hard question only. Second, they suggest replacing medical terminologies in the question of the test set into synonyms to avoid the trivial question which can be solvable with a simple text matching. We use both methods to Relation and Medication datasets to report the performance of every model. For more details on Relation and Medication datasets, please refer to the original paper (Yue et al., 2020). The statistics of training, validation, and test sets on all QA datasets are provided in Table 9.
Named Entity Recognition We use three different domain-specific datasets for evaluating our KALA on NER tasks: CoNLL-2003 (Sang and Meulder, 2003) (News), WNUT-17 (Derczynski et al., 2017) (Social Network Service) and NCBI-Disease (Dogan et al., 2014) (Biomedical). The CoNLL-2003 is constructed from the manually curated 1,393 English news articles, including 301.4k tokens, which has 9 class labels. The WNUT-17 dataset consists of 65,124 emerging and rare entities from social media (e.g., Twitter, Reddit, YouTube, to name a few), which has 13 class la
Table 11: NER dataset statistics. We report the number of contexts (i.e., # Context), with the average length of them (i.e., C. Length) on training, validation, and test sets.
Hyperparameters	News	Medical Textbook
Domain-Adaptive Pre-training (DAPT)
The number of text (by lines)	10M	100k
The number of text (by words)	618M	12.8M
The size of data (by volume)	3.5G	86M
Maximum Sequence Length	384
Batch Size	64
Training Epochs	50
Maximum Steps	12.5k
Optimizer	AdamW
Learning rate	5e-5
Weight Decay	0.01
LR decay Warmup rate	0.06
Half Precision	Yes
NewsQA	Relation
Applied Dataset	CoNLL-2003	Medication
WNUT-17	NCBI-Disease
Table 12: Hyperparameters for DAPT on two domains we used for reporting the performances in the main paper.
bels. The NCBI-Disease dataset consists of the 793 PubMed articles from the biomedical domain, which contains 6,892 disease mentions and 790 disease concepts, and also has 3 class labels. The statistics of training, validation, and test sets are provided in Table 11.