SlowGuess/NAACL_2022_OCR huggingface dataset:
Fedor Moiseev* 1,2 Zhe Dong† 2 Enrique Alfonseca 2 Martin Jaggi 1 EPFL, Switzerland Google, Switzerland
{femoiseev, zhedong, ealfonseca}@google.com, martin.jaggi@epfl.ch
Large language models (LLMs) have demonstrated human-level performance on a vast spectrum of natural language tasks. However, it is largely unexplored whether they can better internalize knowledge from a structured data, such as a knowledge graph, or from text. In this work, we propose a method to infuse structured knowledge into LLMs, by directly training T5 models on factual triples of knowledge graphs (KGs). We show that models pre-trained on Wikidata KG with our method outperform the T5 baselines on FreebaseQA and WikiHop, as well as the Wikidata-answerable subset of TriviaQA and NaturalQuestions. The models pretrained on factual triples compare competitively with the ones on natural language sentences that contain the same knowledge. Trained on a smaller size KG, WikiMovies, we saw  $3 \times$  improvement of exact match score on MetaQA task compared to T5 baseline. The proposed method has an advantage that no alignment between the knowledge graph and text corpus is required in curating training data. This makes our method particularly useful when working with industry-scale knowledge graphs.
Large pre-trained language models, such as BERT (Devlin et al., 2019), GPT-3 (Brown et al., 2020), T5 (Raffel et al., 2019), REALM (Guu et al., 2020) and ERNIE (Sun et al., 2021) have become the state-of-the-art technology for many tasks. They are commonly pre-trained using unstructured text corpora, on tasks such as next word prediction, next sentence prediction (NSP) or masked language modelling (MLM). Especially for T5, self-supervised learning on unlabelled text corpus with MLM has been a common pre-training recipe (Roberts et al., 2020). This is normally followed by a fine-tuning step on the task of interest (Ruder
et al., 2019), although large language models have also proved useful without this task-specific finetuning (Brown et al., 2020).
Beyond the capacity of contextual understanding, human-level language understanding pivots on the knowledge about the world. The world knowledge is often expressed as factual triples (c.f. Ji et al., 2020), in the form of (subject entity, relation, object entity). A knowledge graph (KG) defined by a set of factual triples consists of the subjects and objects as vertices/nodes, and the relations forming the edges connecting them. Most of the large scale KGs (e.g. Wikidata, Vrandecic and Krötzsch, 2014) are stored in triple format.
LLMs demonstrate some capacity of learning world knowledge from the natural text corpus (Roberts et al., 2020), but it is unclear to what degree they are also able to learn and memorize new knowledge directly from structured KG triples, or from text describing them explicitly.
In order to infuse knowledge into a LLM, one option is to generate a textual version of the knowledge base, and apply the standard training objectives, e.g. MLM. This is unfortunately highly nontrivial. One can either align sentences with KG triples, as done in ERNIE (Sun et al., 2021), or generate sentences from triples, as done in KELM (Agarwal et al., 2021). These approaches are unfortunately hard to port to knowledge graphs with different schemas. These processes are also lossy in that not every triple can be aligned or produce a valid sentence, and there is not a good understanding whether this can introduce unnecessary selection biases on top of biases existing in the original KG.
In this work, we propose a method of Knowledge Infusion for Large Language Models (SKILL), where LLMs directly learn from knowledge triples. Experiment results show the checkpoints trained with proposed method on Wikidata KG outperform the T5 baselines on four standard
*Work done during internship at Google.
† Correspondence Author.
Human Language Technologies, pages 1581 - 1588
closed-book question-answering (QA) tasks. With a smaller KG, WikiMovies, the proposed method gain  $3 \times$  exact match score performance improvement on MetaQA task. The models learning directly from knowledge triples performs competitively with the ones with the aligned natural sentences that contain the same amount of knowledge. Being able to learn directly from knowledge triples enables easy addition of structured knowledge into language modeling pre-training.
Previous works that use knowledge graphs to enhance the quality of knowledge-intensive downstream tasks can be divided into two groups: using knowledge graphs at the inference time, and infusing knowledge into the model weights at the pre-training time. The proposed method falls in the latter group.
Explicit usage of knowledge graphs. A retrieval-augmented model is commonly used, in order to retrieve and apply the knowledge from external memories or sources. FILM (Verga et al., 2021) and EaE (Févry et al., 2020) extend Transformer (Vaswani et al., 2017) models with external entity (both FILM and EaE) and fact (FILM) memories. REALM (Guu et al., 2020) is pre-trained to perform reasoning over a large textual knowledge corpus on-the-fly during inference. UniK-QA (Oguz et al., 2020) combines the structured and unstructured information to improve the open-domain QA tasks with a retriever-reader framework. The main difference between the proposed method, SKILL, and retrieval-augmented models is that SKILL doesn't introduce retrieval system or external memories to the model, but it directly embeds knowledge into the model parameters, which introduces no extra cost at inference time.
Knowledge infusion. A common way of parameterized knowledge infusion is to map or convert structured knowledges into natural language text. ERNIE 3.0 (Sun et al., 2021) trains a knowledge-enhanced model on a corpus combining triples and their aligned sentences, by randomly masking relation in a triple or words in a sentence. On the contrary, SKILL trains only on triples.
KnowBert (Peters et al., 2019) incorporates knowledge from Wikipedia and WordNet (Miller, 1995) into a BERT model through entity
embeddings with knowledge-attention and recontextualization mechanism. BERT-MK (He et al., 2020) is a BERT-based model that integrates graph contextual knowledge of a medical KG, which demonstrates the utility of graph-level knowledge. These approaches require entity linking and sentences contextualizing the knowledge graph information.
KG-FiD (Yu et al., 2021) extends the Fusion-in-Decoder model (Izacard and Grave, 2021) with a module that filters and re-ranks passages based on structural connections in knowledge graph between entities described in those passages. In contrast to the SKILL method that we propose, it requires the existence of natural text passages describing each knowledge graph entity, so Wikipedia corpus was used since it naturally provides articles that describe entities.
Heinzerling and Inui (2021) explored the ability of language models to memorize and understand information from knowledge graphs, but used natural language representation of triples based on predefined templates instead of structured representation. Usage of predefined templates significantly limits scalability and therefore only relatively small knowledge graphs were used, such as Google-RE<sup>1</sup>.
In contrast to the new method presented in this paper, all of these approaches require an explicit mapping between the knowledge graph entities or facts and corresponding natural language sentences, which can limit applications to industry-scale knowledge graphs that don't have such a mapping.
Different goals of using knowledge graphs. Besides that, some papers embed knowledge into model weights but pursue different goals rather than improving performance on downstream tasks. COMET (Bosselut et al., 2019) is most similar to our work and trains a commonsense-aware Transformer Language Model by learning to generate loosely structured commonsense descriptions in the natural language given the structured knowledge. Similar to us, it also uses KG triples in surface form as a source for training data, but in contrast to our research, the final goal of COMET is to generate new knowledge instead of utilizing existing ones. Another important difference is the scale: COMET uses Atomic (Sap et al., 2019) and ConceptNet (Speer et al., 2017) Knowledge Graphs
<sup>1</sup>https://ai.googleblog.com/2013/04/50000-lessons-on-how-to-read-relation.html
that are much smaller than Wikidata (Vrandecic and Krötzsch, 2014).
KELM (Agarwal et al., 2021) fine-tunes a T5 model to convert KGs to synthetic natural language sentences to augment existing pre-training corpora. We build our research on top of it and use the KELM dataset to compare structured and natural language representations of knowledge.
There are two components of knowledge infusion for LLMs (SKILL): the corpus and the training method. We introduce the method based on Wiki-data KG, but it can be applied to any other KGs.
Training corpus. We use two corpora with different knowledge representations: Wikidata KG (Vrandecić and Krötzsch, 2014) in triple format, and KELM corpus<sup>2</sup> (Agarwal et al., 2021) as synthetic natural language sentences converted from Wikidata KG. The KELM corpus contains 15,628,486 synthetic sentences. To ensure two corpora share the same knowledge, we take the snapshot of the Wikidata KG used to create the KELM corpus, which contains 35,697,715 triples.
To prevent the degradation of model performance on natural language understanding, we mix the Wikidata corpus or KELM corpus with natural text from C4 (Raffel et al., 2019),  $50:50$ , for the knowledge infusion training data.
Training method. T5 (Raffel et al., 2019) was trained through masked-language modelling with random span corruption on the C4 corpus. Roberts et al. (2020) found that masking salient terms (Guu et al., 2020) in pre-training T5 models, instead of masking random token spans, could significantly improve the performance on downstream tasks, e.g. closed-book QA.
We apply salient span masking for unsupervised learning in our knowledge-infusing training. To mask the same amount of information is for both corpora, the following method is applied. For a knowledge triple, we mask either the subject or object entity. For a KELM sentence, we identify the aligned triple, with details in Appendix A, and mask the full spans corresponding to the subject or object in the triple. The relation tokens are never masked, as there is no robust way to map the abstract relation in knowledge triples to natural
language tokens in KELM sentences. Examples of the inputs for both corpora are in Table 1.
We assess SKILL by training and evaluating the knowledge infused models on closed-book QA tasks, where questions are provided without supporting context and external knowledge.
SKILL pre-training. We apply SKILL on three T5.1.1 pre-trained checkpoints<sup>3</sup>, base, large, and XXL, with sizes of  $\sim 250\mathrm{M}$ ,  $\sim 800\mathrm{M}$  and  $\sim 11\mathrm{B}$  parameters, respectively. For T5.1.1-base and -large, SKILL training is performed for 500K steps with batch size 1024, which translates to  $\sim 7.17$  epochs on Wikidata KG and  $\sim 16.38$  epochs in KELM sentences. For T5.1.1-XXL, the model is trained for 100K steps to finish training in a feasible time.
As baseline we use pre-trained T5 checkpoints of the same size. To make sure that improvements come from knowledge infusion instead of from longer C4 pre-training, we use a second baseline by further training the T5 checkpoints on C4 for half of the aforementioned steps, to match the amount of C4 pre-training used in SKILL.
All the model variations are optimized by AdaFactor (Shazeer and Stern, 2018) with  $10^{-3}$  learning rate and 0.1 dropout rate, the same settings that were used for T5.
Fine-tuning on closed-book QA tasks. We evaluate the checkpoints by fine-tuning on the following QA benchmarks: FreebaseQA (Jiang et al., 2019), WikiHop (Welbl et al., 2018), TriviaQA (Joshi et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019), with the aforementioned hyper-parameters for optimization and 128 batch size. For the benchmarks without a test split, we use the dev split for test, and the last  $10\%$  of train as dev split.
The Exact Match (EM) scores on the test sets are calculated after being fine-tuned for 50K steps for T5.1.1-base and -large models, and 10K steps for -XXL models. All models converged with no noticeable over-fitting according to the EM scores on validation sets.
Wikidata-answerable QA. We found that the majority of the questions in FreebaseQA and Wiki
$^{2}$ Data is available at https://github.com/google-research-datasets/KELM-corpus
3https://goo.gl/t5-checkpoints
Wikidata triple	KELM sentence	Wikidata input	KELM input	Target
("Pulp Fiction",
"award received",
"Palme d'Or")	Quentin Tarantino
won the Palme d'Or in 1994
for Pulp Fiction.	Pulp Fiction,
award received,
[MASK]	Quentin Tarantino
won the [MASK] in 1994
for Pulp Fiction.	Palme d'Or
Table 1: Example inputs for SKILL pre-training with Wikidata and KELM corpora.
Model	FreebaseQA	WikiHop	TQA-matched	TQA	NQ-matched	NQ
dev	test	dev	test	dev	test	dev	test	dev	test	dev	test
base	25.24	27.55	19.09	18.38	31.24	33.55	22.64	22.93	36.64	32.68	25.04	25.48
base + C4	26.19	28.33	19.57	19.36	32.9	34.4	24.54	25.39	36.98	32.03	25.88	25.84
base + WikiKG	26.92	28.38	20.28	20.22	34.21	35.08	24.73	25.77	37.41	33.33	25.51	25.76
base + KELM	26.64	28.15	20.62	19.81	33.64	35.54	25.22	25.75	36.98	32.9	25.31	26.2
large	30.22	32.88	20.92	21.12	36.7	38.09	29.24	30.03	39.22	35.06	27.12	27.15
large + C4	32.55	34.01	22.5	21.51	38.78	40.6	30.32	30.83	39.74	35.5	27.46	28.17
large + WikiKG	33.22	35.29	23.5	23.4	39.19	41.02	29.74	30.47	41.12	35.93	27.38	27.89
large + KELM	32.65	34.16	23.34	22.91	39.45	40.76	30.51	30.65	40.95	35.5	27.67	28.56
XXL	43.67	45.02	24.76	24.8	51.73	53.1	42.44	42.21	46.47	43.72	31	32.27
XXL + C4	42.01	44.14	23.34	22.23	50.59	52.19	40.66	40.99	45.43	40.26	30.35	31.08
XXL + WikiKG	45.22	47.25	27.57	27.65	54.17	54.18	42.55	43.54	49.14	44.37	31.11	32.74
XXL + KELM	45.42	45.9	26.11	26.26	53.65	54.21	42.68	42.95	48.53	44.16	31.79	32.6
Table 2: Exact match scores achieved by fine-tuning the checkpoints on closed-book QA tasks. base, large, XXL represent the corresponding T5.1.1-* checkpoints.  $\star -\mathrm{C4}$  are the checkpoints additionally trained on C4 corpus as discussed in Section 3.  $\star -$ WikiKG and  $\star -$ KELM are the checkpoints trained on Wikidata KG triple corpus and KELM sentence corpus. The best performed checkpoints are in bold. Details about datasets are in Appendix B.
Hop can be answered directly from triples in Wikidata. This is because FreebaseQA was created by matching question-answer pairs with triples in Freebase (Bollacker et al., 2008), most of which was imported into Wikidata (Vrandecic and Krötzsch, 2014). For WikiHop, the questions were generated from Wikidata triples.
However, TriviaQA and NaturalQuestions were created independently of Wikidata, and not every question can be answered using this knowledge base. We found frequent freshness issues, e.g. the golden answer for question "Who is the largest supermarket chain in the UK?" is "Aldi", while today it would be "Tesco". Some other questions can not be answered by WikiData, e.g. "Who, during a radio microphone test in 1984 said, 'I just signed legislation which outlaws Russia forever. The bombing begins in five minutes?'" with the golden answer "Ronald Reagan".
To mitigate this, we created subsets of TriviaQA (TQA) and NaturalQuestions (NQ) that were somewhat more likely to have answers in Wikidata. We selected all the items for which there exist a triple in Wikidata that has the answer either as subject or object, and the other entity in the triple is mentioned in the question. We match the entities by entity name, case-insensitive. We name the Wikidata-aligned version of TQA and NQ as TQA-matched and NQ-matched, respectively. The dataset sizes
of all QA tasks are summarized in Appendix B.
The results for closed-book QA tasks are summarized in Table 2. SKILL pre-trained models show improvements on FreebaseQA, WikiHop, and Wikidata-answerable versions of TriviaQA and NaturalQuestions, but no significant improvement on original TriviaQA and NaturalQuestions. As discussed in previous section, we believe this is due to the misalignment between the datasets and Wikidata.
Models pre-trained on Wikidata KG gives competitive results with ones on KELM sentences. It shows that the triple representation is as good as natural language representation, while being much easier to scale up for larger KG.
For T5.1.1-base and -large, additional pretraining on C4 boosts performance in comparison to the original baseline. For T5.1.1-XXL, this additional pre-training leads to a performance regress. In (Raffel et al., 2019), it is mentioned that training on C4 for multiple times may reduce the performance of a T5 model.
Impact of model size. As shown in Figure 1, SKILL pre-training introduces bigger improvements when applied on larger models. With more than 35M triples in Wikidata KG, it is harder for
Figure 1: Performance improvements on closed-book QA tasks for different model sizes. The improvements are measured by the difference of exact match score  $(\Delta \mathrm{EM})$  between knowledge-infused model trained with Wikidata triples and the baseline trained with C4 corpus.
Dataset	Split	Baseline	+ C4	+ KG
1-hop	dev	24.3	23.12	71.52
test	24.5	23.53	71.47
2-hop	dev	32.05	32.23	33.49
test	32.65	32.78	33.57
3-hop	dev	42.08	39.22	43.79
test	42.31	39.66	43.41
Table 3: Exact match scores achieved by fine-tuning different T5.1.1-large checkpoints on MetaQA task.
smaller size models, e.g. T5.1.1.-base with 300M parameters, to memorize them efficiently. We view this as an encouraging result, suggesting that as model size grows, gains from SKILL pre-training may increase further.
Performance on a smaller KG. The WikiMovies KG (Miller et al., 2016) contains 134, 741 triples. T5.1.1-large should have enough parameters to memorize the KG. We train a T5.1.1-large model on the KG for 100K steps,  $\sim$  380 epochs, with the same hyperparameters as for Wikidata KG. We evaluate the checkpoints with MetaQA (Zhang et al., 2018) benchmark that was constructed over WikiMovies KG. The benchmark contains 3 different sub-tasks: 1-hop QA (e.g. "What films does Paresh Rawal appear in?"), 2-hop QA (e.g. "Who are the directors of the films written by Laura Kerr?"), 3-hop QA (e.g. "Who directed the movies written by the writer of Millennium Actress?").
The results in Table 3 demonstrate the effectiveness of SKILL pre-training, when it's possible to memorize the whole knowledge graph.
As 1-hop questions are supported by single triples in the WikiMovies KG, a  $3 \times$  improvement on EM score is observed for the sub-task. In order to answer 2/3-hop questions it is not enough to memorize the triples, the model needs to be able
to reason with them. This requires a better understanding of the graph structure. Training with single triples may not be enough, and the observed improvement is notably smaller. The performance could be further improved by representing more explicitly the graph structure in the training data, which we leave for future work.
We proposed a method to directly infuse knowledge from knowledge graphs into T5 models through pre-training. Empirical results show that T5 can learn directly from structured data and apply the learned knowledge to improve closed-book QA results. We also demonstrated that the models pre-trained on factual triples perform competitively with the ones on natural language sentences that contain the same knowledge. By enabling knowledge infusion directly from triples, this method can be very easily applied to industry-scale KGs.
6 Ethical and Broader Impact
In this work, we are introducing a new method to pre-train a well known natural language understanding model, T5, on the full corpora of public knowledge graphs. To the best of our knowledge, the method will not introduce extra bias to either the model or the dataset beyond the one potentially inherited from Wikidata (Vrandecic and Krötzsch, 2014) and WikiMovies (Miller et al., 2016) knowledge graphs. On the other hand, through knowledge fusion pre-training introduced in this work, a language model will be able to learn factual information to improve the quality of parameterized knowledge embedded in the model, which is demonstrated by improvements on various closed-book question-answering tasks. The proposed method and recipe will provide positive impact to the natural language processing community and help to improve the factualness in pre-trained large language model checkpoints.
Limitations. A factual triple is the basic ingredient of a knowledge graph. However, as a semantic network, the graph structure of a knowledge graph describes how the factual triples are connected. This information is not easy to directly represent by random set of triples. We leave the exploration of how to infuse the information implied by the graph structure for future work. We expect that this will further improve the results, especially for multi-hop question-answering tasks.
Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami Al-Rfou. 2021. Knowledge graph based synthetic corpus generation for knowledge-enhanced language model pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3554-3565, Online. Association for Computational Linguistics.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: A collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, SIGMOD '08, page 1247-1250, New York, NY, USA. Association for Computing Machinery.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense transformers for automatic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762-4779, Florence, Italy. Association for Computational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, and Tom Kwiatkowski. 2020. Entities as experts: Sparse memory access with entity supervision. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4937-4951, Online. Association for Computational Linguistics.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International Conference on Machine Learning, pages 3929-3938. PMLR.