SlowGuess/NAACL_2021_OCR huggingface dataset:
The REPLACED and AUGMENTED models are evaluated using both the raw Wikidata triples and the generated sentences. Wikidata triples are grouped by subject entity to form Triple Documents and KELM Corpus sentences are also grouped by subject entity to form KELM Corpus Documents (§4.2). The model is pre-trained for 200k steps with the CC-News pre-training corpus in all cases with default hyperparameters.
ORIGINAL For NQ and WQ, we fine-tuned the pre-trained REALM on the respective training splits. While we were able to reproduce the accuracy on WQ, the accuracy on NQ was  $\sim 1.5\%$  absolute less than the reported accuracy (row 1&2 in Table 7). For LAMA probe, we first evaluated the pre-trained REALM, reporting the results on the different sub-corpora in Table 6 (row Wikipedia under REALM). Even the ORIGINAL REALM model shows substantial improvement over prior models. The ability of REALM to access the corpus documents during inference not only make it interpretable but also better on the knowledge intensive tasks. It obtains an accuracy of  $67.36\%$  on Google-RE,  $68.18\%$  on T-REx and  $27.96\%$  on
3https://code.google.com/archive/p/ relation-extraction-corpus/
Google-RE	TREx	Squad	Concept Net
DOB	POB	POD	Total	1-1	N-1	N-M	Total
Elmo 5.5B (Peters et al., 2018)	0.10	7.50	1.30	3.00	13.10	6.50	7.40	7.10	4.30	6.20
Tranformer-XL (Dai et al., 2019)	0.90	1.10	2.70	1.60	36.50	18.00	16.50	18.30	3.90	5.70
BERT-large (Devlin et al., 2019)	1.40	16.10	14.00	10.50	74.50	34.20	24.30	32.30	17.40	19.20
REALM
ORIGINAL Wikipedia	49.06	79.56	64.13	67.36	55.81	69.54	66.98	68.18	27.96	4.78
REPLACED Triple Documents	69.46	61.64	53.01	63.03	49.30	62.34	53.12	58.43	18.09	4.27
KELM Documents	68.91	61.37	53.79	62.81	49.41	61.60	52.50	57.76	19.07	4.26
AUGMENTED Wikipedia + Triple Documents	71.60	80.92	69.89	76.32	57.20	69.96	67.86	68.80	29.93	4.81
Wikipedia + KELM Documents	76.75	83.92	74.86	80.30	57.84	70.33	68.09	69.13	31.57	5.25
Table 6: Accuracy on LAMA probe. Pretaining corpus is CCnews and the retrieval corpus changed for REALM.
REALM Retrieval Corpus	NQ	WQ
ORIGINAL
Wikipedia (reported)	40.40	40.70
Wikipedia (rerun)	38.84	40.80
REPLACED
Triple Documents	21.14	42.54
KELM Documents	22.58	41.19
AUGMENTED
Wikipedia + Triple Documents	40.28	42.91
Wikipedia + KELM Documents	41.47	43.90
Table 7: Exact Match (EM) accuracy of REALM on NQ and WQ. Pretraining corpus used is CC-News.
SQuAD. In comparison, the reported accuracy for BERT (Devlin et al., 2019) is  $10.50\%$  on GoogleRE,  $32.30\%$  on T-REx and  $17.40\%$  on SQuAD. BERT performs better on 1-1 T-REx relations with  $74.50\%$  accuracy as compared to REALM with  $55.81\%$  accuracy. However, this group consists of only two relations; capital and capital of. BERT also has better performance than REALM on the ConceptNet subcorpus. On inspection of some of the queries in ConceptNet, we found the questions to be vague and possibly hard for even humans. For example, Raven can and Time is
REPLACED The REPLACED model which uses only KELM Corpus Documents, performs better than the ORIGINAL model on WQ but the accuracy is much lower on NQ (rows 3&4 in Table 7). This can be attributed to the nature of the datasets-WQ is a KG-based dataset whereas NQ consists of real queries issued to Google. On LAMA (rows 2&3 under REALM in Table 6), the performance is lower than the ORIGINAL model but much higher than BERT. Both Triple Documents and KELM Corpus Documents have similar performance. When using just the KG, the format doesn't matter. However, a system trained on raw triples may not generalize for tasks where sentence structure is important.
AUGMENTED We observe improvements on all the datasets (last two rows of Tables 6&7) with the AUGMENTED model which uses both the Wikipedia text and the KELM Documents. There is an absolute gain of  $2.63\%$  and  $3.10\%$  on NQ and WQ respectively over the ORIGINAL model. Similarly, there is an absolute gain of  $12.94\%$ ,  $0.95\%$ ,  $3.61\%$  and  $0.47\%$  on Google-RE, T-REx, SQuAD and ConceptNet in LAMA respectively. Unlike the REPLACED model, the improvement is higher when the generated sentences in KELM Corpus are added instead of the raw Wikidata triples, confirming the effectiveness of verbalization of KG into natural language sentences. Wikipedia is the dominant corpus with 2B words whereas KELM corpus sentences are succinct with a total of 286M words ( $\S 4.2$ ) so it is likely the learned representations favour the Wikipedia format which is natural language sentences.
We expect augmenting other retrieval-based models such as DPR (Karpukhin et al., 2020) and RAG (Lewis et al., 2020) with the KELM corpus should also improve their performance, given that their enhancements are orthogonal to our contribution. Moreover, we note that our augmented corpus represents a scalable strategy for future QA systems; by adding only  $14\%$  more tokens to the original REALM model we outperform huge and computationally expensive models such as (Roberts et al., 2020) (11B parameters) on NQ  $(35.20 \rightarrow 41.47)$  and WQ  $(42.80 \rightarrow 43.90)$ . Wikipedia is the dominant corpus with 2B words whereas KELM corpus sentences are succinct with a total of 286M words ( $\S 4.2$ ) so it is likely the learned representations favour the Wikipedia format which is natural language sentences.
We inspected the errors of the AUGMENTED model with KELM Documents on LAMA. Apart
from real errors where the prediction is actually incorrect, there were some false errors that can be broadly classified into three categories-
1. Ambiguous Query: e.g. In "X was born in ______", the answer could be the year or the place of birth but only one of them is acceptable depending on the subcorpus.
2. Incomplete Answer Set: e.g. In “Konstantin Mereschkowski had a career as ______”, the gold target is biologist and the prediction is botanist but both should be correct.
3. Answer granularity: The prediction is correct but more specific. e.g. In "On the CPI scale, Kenya ranks ______", the gold answer is low but the prediction is 101, which is in fact correct.
Data-to-Text Generation Data-to-Text Generation has several benchmark datasets with slightly different objectives—WebNLG (Gardent et al., 2017) to convert a group of triples to text, E2ENLG (Dusek et al., 2018) to convert database key-value pairs or pictures to text, WikiBio (Lebret et al., 2016) for biography generation from text, Wiseman et al. (2017) for text describing score statistics tables of basketball games, both ToTTo (Parikh et al., 2020) and DART (Radev et al., 2020) to generate text given a table and relevant highlighted cells. Many systems (van der Lee et al., 2018; Castro Ferreira et al., 2019; Shimorina and Gardent, 2018) have been developed and evaluated on these datasets, such as graph transformers over structured data (Koncel-Kedziorski et al., 2019), latent templates for interpretability (Wiseman et al., 2018) and text-to-text generation with T5 (Kale, 2020).
KG-Text alignment T-REx (Elsahar et al., 2018) is a widely used Text-KG aligned corpus, built using systems such as coreference resolution and predicate linkers (details in §2.1.1). Logan et al. (2019) and Chen et al. (2020) also created an aligned corpus using Wikipedia hyperlinks and coreference resolution. (details on comparison in §2.1.2). In contrast, we use alias-based heuristics coupled with source text selection constraints to generate a corpus of 16M triples aligned with 8M sentences. Lastly, open information extraction i.e. automatic KG construction from text (Etzioni et al., 2008; Angeli et al., 2015; Clancy et al., 2019) inherently create such a corpus but these works generally do not release the extracted KG triples.
Incorporating KGs Most prior works on incorporating KG with text often learn KG entity representations and add them to the mention spans linked to the entity (Peters et al., 2019; Yu et al., 2020; Févry et al., 2020) or create subgraphs relevant to the query that are expanded with text in the embedding space (Logan et al., 2019; Sun et al., 2019; Xiong et al., 2019). Some others incorporate additional modules. Verga et al. (2020) extend Févry et al. (2020) by adding a triple memory with (subject, relation) encoding as the key and the object encoding as the value. Das et al. (2017) use universal schema (Riedel et al., 2013) that embeds text and KGs in a shared space for their integration. K M et al. (2018) learn a single representation for all the triples mentioned in a sentences during pre-training and update it further in task-specific finetuning. In contrast, we convert the KG into text and use it to augment the pre-training data.
The KELM corpus sentences covers all facts in the KG but the generated sentences are limited to a given entity and its direct relations to other entities. For example, given the triples (X, child, Y) and (Y, child, Z), it does not contain "Z is a grandchild of X". More complex sentences could be generated by incorporating multi-hop relations in the KG. Recent work has also shown promising results on generating multilingual text from English triples (Castro Ferreira et al., 2020; Agarwal et al., 2020). Our proposed approach can be applied to generate a multilingual corpus of facts in various languages using English Wikidata.
In this paper, we converted an entire KG (Wikidata) to natural text (KELM Corpus), tackling various challenges over verbalizing domain-specific benchmark datasets. We further showcase that KG verbalization can be used to integrate KGs and natural text corpora by including the verbalized KG as additional pre-training data. We augment a retrieval-based language model with the generated synthetic KELM corpus as a retrieval corpus. We evaluated the augmented model on open domain QA and a knowledge probe, showing improvements on both. The KELM Corpus is publicly available at https://github.com/google-research-datasets/KELM-corpus.
We thank William Woods, Jonni Kanerva, Tania Rojas-Esponda, Jianmo Ni, Aaron Cohen and Itai Rolnick for rating the synthetic corpus sample for human evaluation. We also thank Kelvin Guu for his valuable feedback on the paper.
Oshin Agarwal, Mihir Kale, Heming Ge, Siamak Shakeri, and Rami Al-Rfou. 2020. Machine translation aided bilingual data-to-text generation and semantic parsing. In Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+), pages 125-130, Dublin, Ireland (Virtual). Association for Computational Linguistics.
Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D. Manning. 2015. Leveraging linguistic structure for open domain information extraction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 344-354, Beijing, China. Association for Computational Linguistics.
Jonathan Berant, Ido Dagan, and Jacob Goldberger. 2010. Global learning of focused entailment graphs. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1220-1229, Uppsala, Sweden. Association for Computational Linguistics.
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in neural information processing systems, pages 4349-4357.
Thiago Castro Ferreira, Claire Gardent, Nikolai Ilinykh, Chris van der Lee, Simon Mille, Diego Moussallem, and Anastasia Shimorina. 2020. The 2020 bilingual, bi-directional WebNLG+ shared task: Overview and evaluation results (WebNLG+ 2020). In Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+), pages 55-76, Dublin, Ireland (Virtual). Association for Computational Linguistics.
Thiago Castro Ferreira, Chris van der Lee, Emiel van Miltenburg, and Emiel Krahmer. 2019. Neural data-to-text generation: A comparison between pipeline and end-to-end architectures. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 552-562, Hong
Kong, China. Association for Computational Linguistics.
Wenhu Chen, Yu Su, Xifeng Yan, and William Yang Wang. 2020. Kgpt: Knowledge-grounded pretraining for data-to-text generation. arXiv preprint arXiv:2010.02307.
Ryan Clancy, Ihab F. Ilyas, and Jimmy Lin. 2019. Scalable knowledge graph construction from text collections. In Proceedings of the Second Workshop on Fact Extraction and VERIFICATION (FEVER), pages 39-46, Hong Kong, China. Association for Computational Linguistics.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860.
Rajarshi Das, Manzil Zaheer, Siva Reddy, and Andrew McCallum. 2017. Question answering on knowledge bases and text using universal schema and memory networks. arXiv preprint arXiv:1704.08384.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Ondrej Dušek, Jekaterina Novikova, and Verena Rieser. 2018. Findings of the e2e nlg challenge. arXiv preprint arXiv:1810.01170.
Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018. T-REx: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Oren Etzioni, Michele Banko, Stephen Soderland, and Daniel S Weld. 2008. Open information extraction from the web. Communications of the ACM, 51(12):68-74.
Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, and Tom Kwiatkowski. 2020. Entities as experts: Sparse memory access with entity supervision. arXiv preprint arXiv:2004.07202.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. The WebNLG challenge: Generating text from RDF data. In Proceedings of the 10th International Conference on Natural Language Generation, pages 124-133, Santiago de Compostela, Spain. Association for Computational Linguistics.
E. Goldberg, N. Driedger, and R. I. Kittredge. 1994. Using natural-language processing to produce weather forecasts. IEEE Expert, 9(2):45-53.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Realm: Retrievalaugmented language model pre-training. arXiv preprint arXiv:2002.08909.
Annervaz K M, Somnath Basu Roy Chowdhury, and Ambedkar Dukkipati. 2018. Learning beyond datasets: Knowledge graph augmented neural networks for natural language processing. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 313-322, New Orleans, Louisiana. Association for Computational Linguistics.
Mihir Kale. 2020. Text-to-text pre-training for data-to-text tasks. arXiv preprint arXiv:2005.10433.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wentau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906.
Rik Koncel-Kedziorski, Dhanush Bekal, Yi Luan, Mirella Lapata, and Hannaneh Hajishirzi. 2019. Text Generation from Knowledge Graphs with Graph Transformers. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2284-2293, Minneapolis, Minnesota. Association for Computational Linguistics.
Karen Kukich. 1983. Design of a knowledge-based report generator. In 21st Annual Meeting of the Association for Computational Linguistics, pages 145-150, Cambridge, Massachusetts, USA. Association for Computational Linguistics.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics.
Rémi Lebret, David Grangier, and Michael Auli. 2016. Neural text generation from structured data with application to the biography domain. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1203-1213, Austin, Texas. Association for Computational Linguistics.
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation
for knowledge-intensive nlp tasks. arXiv preprint arXiv:2005.11401.
Robert Logan, Nelson F. Liu, Matthew E. Peters, Matt Gardner, and Sameer Singh. 2019. Barack's wife hillary: Using knowledge graphs for fact-aware language modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5962-5971, Florence, Italy. Association for Computational Linguistics.
Thomas Manzini, Lim Yao Chong, Alan W Black, and Yulia Tsvetkov. 2019. Black is to criminal as caucasian is to police: Detecting and removing multiclass bias in word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 615-621, Minneapolis, Minnesota. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Ankur P Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. Todo: A controlled table-to-text generation dataset. arXiv preprint arXiv:2004.14373.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 43-54, Hong Kong, China. Association for Computational Linguistics.
Fabio Petroni, Tim Rocttäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China. Association for Computational Linguistics.
Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Nazneen Fatema Rajani, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, et al. 2020. Dart: Open-domain structured data record to text generation. arXiv preprint arXiv:2007.02871.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M. Marlin. 2013. Relation extraction with matrix factorization and universal schemas. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 74-84, Atlanta, Georgia. Association for Computational Linguistics.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418-5426, Online. Association for Computational Linguistics.
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3407-3412, Hong Kong, China. Association for Computational Linguistics.
Anastasia Shimorina and Claire Gardent. 2018. Handling rare items in data-to-text generation. In Proceedings of the 11th International Conference on Natural Language Generation, pages 360-370, Tilburg University, The Netherlands. Association for Computational Linguistics.
Robyn Speer and Catherine Havasi. 2012. Representing general relational knowledge in Concept-Net 5. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pages 3679-3686, Istanbul, Turkey. European Language Resources Association (ELRA).
Haitian Sun, Tania Bedrax-Weiss, and William Cohen. 2019. PullNet: Open domain question answering with iterative retrieval on knowledge bases and text. In Proceedings of the 2019 Conference on Empirical