SlowGuess/NAACL_2021_OCR huggingface dataset:
supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada. Association for Computational Linguistics.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wentau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906.
Nora Kassner and Hinrich Schütze. 2020. Bertknn: Adding a knn search component to pretrained language models for better qa. arXiv preprint arXiv:2005.00766.
John E Laird, Christian Lebiere, and Paul S Rosenbloom. 2017. A standard model of the mind: Toward a common computational framework across artificial intelligence, cognitive science, neuroscience, and robotics. *Ai Magazine*, 38(4):13-26.
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086-6096.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020a. Retrieval-augmented generation for knowledge-intensive nlp tasks. arXiv preprint arXiv:2005.11401.
Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2020b. Question and answer test-train overlap in open-domain question answering datasets. arXiv preprint arXiv:2008.02637.
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation embeddings for knowledge graph completion. In Twenty-ninth AAAI conference on artificial intelligence.
Robert Logan, Nelson F. Liu, Matthew E. Peters, Matt Gardner, and Sameer Singh. 2019. Barack's wife hillary: Using knowledge graphs for fact-aware language modeling. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Todor Mihaylov and Anette Frank. 2018. Knowledgeable reader: Enhancing cloze-style reading comprehension with external commonsense knowledge. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1:
Long Papers), pages 821-832, Melbourne, Australia. Association for Computational Linguistics.
Alexander Miller, Adam Fisch, Jesse Dodge, Amir Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly reading documents. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1400-1409.
Allen Newell, J. C. Shaw, and Herbert A. Simon. 1959. Report on a general problem-solving program. In Proceedings of the International Conference on Information Processing.
Allen Newell and Herbert Simon. 1956. The logic theory machine-a complex information processing system. IRE Transactions on information theory, 2(3):61-79.
H Leo H de Penning, Artur S d'Avila Garcez, Luis C Lamb, and John-Jules C Meyer. 2011. A neuralsymbolic cognitive agent for online learning and reasoning. In Twenty-Second International Joint Conference on Artificial Intelligence.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237.
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Fabio Petroni, Tim Roktaschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019. Language models as knowledge bases? arXiv preprint arXiv:1909.01066.
Gadi Pinkas. 1991. Symmetric neural networks and propositional logic satisfiability. *Neural Computation*, 3(2):282-291.
Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2019. Bert is not a knowledge base (yet): Factual knowledge vs. name-based reasoning in unsupervised qa. arXiv preprint arXiv:1911.03681.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
Hongyu Ren, Weihua Hu, and Jure Leskovec. 2020. Query2box: Reasoning over knowledge graphs in vector space using box embeddings. International Conference on Learning Representations.
Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M Marlin. 2013. Relation extraction with matrix factorization and universal schemas. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 74-84.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910.
Haitian Sun, Andrew O Arnold, Tania Bedrax-Weiss, Fernando Pereira, and William W Cohen. 2020. Guessing what's plausible but remembering what's true: Accurate neural reasoning for question-answering. Advances in neural information processing systems.
Haitian Sun, Tania Bedrax-Weiss, and William Cohen. 2019. Pullnet: Open domain question answering with iterative retrieval on knowledge bases and text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2380-2390.
Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William Cohen. 2018. Open domain question answering using early fusion of knowledge bases and text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4231-4242.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. Bert rediscovers the classical nlp pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601.
Théo Trouillon, Christopher R Dance, Éric Gaussier, Johannes Welbl, Sebastian Riedel, and Guillaume Bouchard. 2017. Knowledge graph completion via complex tensor factorization. The Journal of Machine Learning Research, 18(1):4735-4772.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Patrick Verga, David Belanger, Emma Strubell, Benjamin Roth, and Andrew McCallum. 2016. Multilingual relation extraction using compositional universal schema. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 886-896.
Patrick Verga, Arvind Neelakantan, and Andrew McCallum. 2017. Generalizing to unseen entities and entity pairs with row-less universal schema. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 613-622.
Denny Vrandecic and Markus Krötzsch. 2014. Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57(10):78-85.
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Cuihong Cao, Daxin Jiang, Ming Zhou, et al. 2020a. K-adapter: Infusing knowledge into pre-trained models with adapters. arXiv preprint arXiv:2002.01808.
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2020b. K-adapter: Infusing knowledge into pre-trained models with adapters.
Dirk Weissenborn, Tomasz Kocisky, and Chris Dyer. 2017. Dynamic integration of background knowledge in neural nlu systems.
Jason Weston, Sumit Chopra, and Antoine Bordes. 2014. Memory networks. arXiv preprint arXiv:1410.3916.
Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov. 2019. Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model. arXiv preprint arXiv:1912.09637.
Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015. Semantic parsing via staged query graph generation: Question answering with knowledge base. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1321-1331, Beijing, China. Association for Computational Linguistics.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. Ernie: Enhanced language representation with informative entities. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Coterell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender bias in contextualized word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 629-634, Minneapolis, Minnesota. Association for Computational Linguistics.
A.1.1 Evaluation Data Statistics
For WebQuestionsSP, we mapped question entities and answer entities to their Wikidata ids.  $87.9\%$  of the questions are answerable by at least one answer entity that is mappable to Wikidata. For all questions in FreebaseQA there exists at least one relational path in Freebase between the question entity  $e_i$  and the answer  $e_{\mathrm{ans}}$ . The path must be either a one-hop path, or a two-hop path passing through a mediator (CVT) node, and is verified by human raters.  $72\%$  of the question entities and  $80\%$  of the answer entities are mappable to Wikidata, and  $91.7\%$  of the questions are answerable by at least one answer entity that is mappable to Wikidata.
Full Dataset	Wikidata Answerable
FreebaseQA	Train	20358	12535
Dev	3994	2464
Test	3996	2440
WebQuestionsSP	Train	2798	1388
Dev	300	153
Test	1639	841
Table 7: Dataset stats. Number of examples in train, dev, and test splits for our three different experimental setups. Full are the original unaltered datasets. Wikidata Answerable keeps only examples where at least one question entity and answer entity are mappable to Wikidata and there is at least one fact between them in our set of facts.
A.1.2 Pretraining Data Details
FILM is pretrained on Wikipedia and Wikidata using the same data from Févry et al. (2020). Text in Wikipedia is chunked into 128 token pieces. To compute the entity-linking loss  $\text{loss}_{ent}$ , we use as training data entities linked to the 1 million most frequently linked-to Wikidata entities. Text pieces without such entities are dropped. This results in 30.58 million text pieces from Wikipedia. As described in §2.1, we generate  $n$  training examples from a piece of text containing  $n$  entity mentions, where each mention serves as the masked target for its corresponding example, and other entity mentions in the example are treated as context entities<sup>10</sup>. This conversion results in 85.58 million
pre-training examples. The knowledge base  $\mathcal{K}$  is a subset of Wikidata that contains all facts with subject and object entity pairs that co-occur at least 10 times on Wikipedia pages.[11] This results in a KB containing 1.54 million KB triples from Wikidata (or 3.08 million if reverse triples are included). Below, this is called the full setting of pretraining—we will also train on subsets of this example set, as described below. We pretrain the model for 500,000 steps with the batch size 2048, and we set  $k = 1^{12}$  in the  $\mathrm{TOP}_k$  operation for fact memory access.
A.2.1 Pretraining
FILM is jointly trained to predict context entities and the masked entity. Context entities are predicted using the contextual embeddings described in §2.3; intermediate supervision with oracle entity linking labels is provided in the entity memory access step for context entities; the masked entity is predicted using the knowledge-enhanced contextual embeddings (§2.5); and distant supervised fact labels are also provided at training time. The final training loss is the unweighted sum of the four losses:
$$
\text {l o s s} _ {\text {p r e t r a i n}} = \text {l o s s} _ {\text {e n t}} + \text {l o s s} _ {\text {c t x}} + \text {l o s s} _ {\text {f a c t}} + \text {l o s s} _ {\text {a n s}}
$$
A.2.2 Finetuning on Question Answering
In the Open-domain Question Answering task, questions are posed in natural language, e.g. "Where was Charles Darwin born?", and answered by a sequence of tokens, e.g. "United Kingdom". In this paper, we focus on a subset of open-domain questions that are answerable using entities from a knowledge base. In the example above, the answer "United Kingdom" is an entity in Wikidata whose identity is Q145.
We convert an open-domain question to an input of FILM by appending the special [MASK] token to the end of the question, e.g. {‘Where’, ‘was’, ‘Charles’, ‘Darwin’, ‘born’, ‘?’, [MASK]}. The task is to predict the entity named by mask. Here, “Charles Darwin” is a context entity, which is also referred to as question entity in the finetuning QA task.
At finetuning time, entity embeddings  $\mathbf{E}$  and relation embeddings  $\mathbf{R}$  are fixed, and we finetune
This leads to more KB triples than entity pairs, since a pair of entities can be connected by more than one relation.
<sup>12</sup>We experimented with other values of  $k$  during fine tuning and evaluation but did not observe significant differences.
<sup>10</sup>We mask context entities randomly with probability .15
all transformer layers and the four transformation matrices:  $\mathbf{W}_a$ ,  $\mathbf{W}_b$ ,  $\mathbf{W}_e$ ,  $\mathbf{W}_f$ . Parameters are tuned to optimize unweighted sum of the fact memory retrieval loss  $\mathrm{loss}_{\mathrm{fact}}$  and the final answer prediction loss  $\mathrm{loss}_{\mathrm{ans}}$ . If multiple answers are available, the training label  $\mathbb{I}_{e_{\mathrm{ans}}}$  becomes a  $k$ -hot vector uniformly normalized across the answers.
$$
\text {l o s s} _ {\text {f i n e t u n e}} = \text {l o s s} _ {\text {f a c t}} + \text {l o s s} _ {\text {a n s}}
$$
The number of Base parameters includes the encoder and (where applicable) decoder transformer parameters derived from the original papers. We exclude token embeddings in this count following prior work. The Memory parameter count for DPR, RAG, and FID includes the number of parameters required to cache and index the full 26 million passage wikipedia corpus with dimension 768 used by those models. For EaE, the Memory is for the entity embedding matrix. For FILM it is both the entity embedding matrix and the cached fact embedding matrix comprised of the 1.7 million precomputed triple embeddings.