SlowGuess/NAACL_2021_OCR huggingface dataset:
Oshin Agarwal*1 Heming Ge² Siamak Shakeri² Rami Al-Rfou²
<sup>1</sup> University of Pennsylvania <sup>2</sup>Google Research
oagarwal@seas.upenn.edu, {hemingge, siamaks, rmyeid}@google.com
Prior work on Data-To-Text Generation, the task of converting knowledge graph (KG) triples into natural text, focused on domain-specific benchmark datasets. In this paper, however, we verbalize the entire English Wikipedia KG, and discuss the unique challenges associated with a broad, open-domain, large-scale verbalization. We further show that verbalizing a comprehensive, encyclopedic KG like Wikidata can be used to integrate structured KGs and natural language corpora. In contrast to the many architectures that have been developed to integrate these two sources, our approach converts the KG into natural text, allowing it to be seamlessly integrated into existing language models. It carries the further advantages of improved factual accuracy and reduced toxicity in the resulting language model. We evaluate this approach by augmenting the retrieval corpus in a retrieval language model and showing significant improvements on the knowledge intensive tasks of open domain QA and the LAMA knowledge probe.
Data-To-Text Generation (Kukich, 1983; Goldberg et al., 1994) involves converting knowledge graph (KG) triples of the form (subject, relation, object) into a natural language sentence(s). There are many standard datasets for this task such as WebNLG (Gardent et al., 2017) and many systems have been developed to improve performance on these datasets. However, to the best of our knowledge, no prior work has attempted to verbalize a full knowledge graph. Verbalizing a full KG has additional challenges over small benchmark datasets, such as entity and relation coverage and the lack of grouped sets of triples that can produce coherent sentences together. In this paper, we convert the English Wikidata KG (Vrandecic and Krötzsch, 2014) into natural language text (Figure 1). The
Figure 1: An example of generating text from KG. First, the entity subgraphs on the left are created and then converted to the sentence on the right.
generated corpus, which we call the KELM Corpus, consists of  $\sim 18\mathrm{M}$  sentences spanning  $\sim 45\mathrm{M}$  triples with  $\sim 1500$  distinct relations. For training the verbalization system, we also create an English Wikidata KG-Wikipedia Text aligned corpus consisting of a variety of entities such as dates and numerical quantities.
We evaluate the quality of the generated corpus through human evaluation of a random sample. We further showcase the utility of this corpus in language model pre-training. Text represents a limited coverage of the world knowledge. Therefore, we expect the language models to be restricted to facts that are expressed in natural language. Moreover, facts may not be expressed as explicitly in text as they are in KGs, and the variability in the quality of text can eventually cause biases in the resulting models (Bolukbasi et al., 2016; Sheng et al., 2019; Manzini et al., 2019). Building models that handle structured data and free form text seamlessly has been a long sought-after goal. However, their integration is challenging due to different structural formats. KG verbalization provides a simple way to integrate KGs with natural text. We illustrate this by augmenting the REALM (Guu et al., 2020) retrieval corpus with the KELM Corpus. We evaluate
*Work done during internship at Google
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3554-3565 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 2: Pipelines for training the TEKGEN model and generating the KELM corpus. In Step  $①$ , KG triples are aligned with Wikipedia text using distant supervision. In Steps  $②$  &  $③$ , T5 is fine-tuned sequentially first on this corpus, followed by a small number of steps on the WebNLG corpus, In Step  $④$ , BERT is fine-tuned to generate a semantic quality score for generated sentences w.r.t. triples. Steps  $②$ ,  $③$  &  $④$  together form TEKGEN. To generate the KELM corpus, in Step  $⑤$ , entity subgraphs are created using the relation pair alignment counts from the training corpus generated in step  $①$ . The subgraph triples are then converted into natural text using TEKGEN.
the augmented system on the LAMA knowledge probe (Petroni et al., 2019) and open domain QA and show improvements on both. Through ablation experiments where we augment the retrieval corpus with the raw triples instead, we further confirm the effectiveness of verbalization. Our contributions are as follows -
- TEKGEN (Text from KG Generator): A data-to-text sequence-to-sequence model for verbalizing an entire KG
- TEKGEN training corpus: Text-KG aligned corpora with a wide variety of relations including dates and quantities
- KELM Corpus,  ${}^{1}$  (Corpus for Knowledge-Enhanced Language Model Pre-training): A large-scale synthetic corpus of Wikidata KG as natural text
- Data-to-text generation as a method to integrate KGs with textual pre-training corpora, showing improvements on open domain QA and LAMA probe with the augmented model
2 TEKGEN
One of the challenges in converting an entire KG to text is the wide variety of entities and relations. Wikidata consists of  $\sim 6\mathbf{M}$  entities and  $\sim 1500$  relations. In comparison, the WebNLG dataset has  $\sim 600$  entities and  $\sim 20$  relations. In this section, we discuss the various components of TEKGEN, also illustrated in Figure 2 -
1. Create a large yet noisy training dataset using distant supervision.
2. Sequentially fine-tune T5, first on the dataset from step 1 for improved coverage, then on a small clean dataset for reduced hallucination.
3. Build a filter for the generated text based on its semantic quality w.r.t. the KG triples.
2.1 Training Dataset
We first create training data using distant supervision by aligning Wikidata triples to Wikipedia text (see Figure 3).
2.1.1 KG-Text Alignment
For each entity, we constrain the candidate sentences to the root section of its Wikipedia page
$^{1}$ Both the TEKGEN training corpus and the KELM corpus are available at https://github.com/google-research-datasets/KELM-corpus
Figure 3: KG-Text alignment algorithm.
because this section generally describes the relations of the subject entity with other entities. For each sentence in this section, we match all triples that have this entity as the subject. A triple is said to match if any alias of the object entity occurs in the sentence. We do not match relations to text as there are too many ways to express them. Constraining to the subject entity's page and root section generally ensures that the relation is expressed in the sentence if it mentions the object entity. Each triple can align to multiple sentences and each sentence can have multiple triples aligned to it. If any alias of the subject entity occurs in the given sentence, the sentence is selected as is, else the first animate third-person personal or possessive pronoun is replaced by the subject entity's canonical name. The pronoun replacement heuristic also works well because of this constraint. All triples aligned to a given sentences are combined together as a single example.
Alignment statistics are shown in Table 1 and some alignment examples are shown in Table 2. There are a total of  $\sim 45\mathrm{M}$  triples,  $\sim 35\%$  of which were aligned to sentences. This results in  $\sim 8\mathrm{M}$  examples, covering  $\sim 42\%$  of the relations.
Note that each sentence in the aligned corpus is matched to triples with a common subject entity. While this results in some noise, such errors should be small due to the constraint that the text is the root section of the subject entity page. This constraint allows us to maintain the same property of common subject entity as the entity subgraph used in inference (§3). It also simplified the alignment process, removing the need to match relations to text. In comparison, the T-REx (Elsahar et al., 2018) corpus does not have this noise due the use of typical NLP pipeline with coreference resolution and predicate linking. However, it suffers from
Total KG triples	45,578,261
Triples aligned	16,090,457
Total sentences aligned	7,978,814
Total KG relations	1,522
Relations aligned	663
Table 1: KG-Text alignment statistics.
errors due to entity linking and incorrect entailment, which are unlikely in our corpus due to this constraint.
2.1.2 Types of Triples
We extract several types of triples, each of which have slightly different matching techniques. Other alignment corpora built using Wikipedia hyperlinks (Chen et al., 2020; Logan et al., 2019) would miss many of these triples with entities without Wikipedia pages such as quantities, dates and certain occupations, and hence relations such as date of birth, publication year and distance from Earth.
1. Object entity with a Wikipedia page: These are aligned by string matching all aliases. (e.g. Barack Obama)
2. Object entity without a Wikipedia page: These are also aligned by matching all aliases. (e.g. skateboarder, professional wrestler)
3. Object entity is a quantity: They have two components - Amount and Units. Units are also entities in the KG and have aliases. We concatenate the amount with each of the unit's aliases for matching (e.g.  $16\mathrm{km/hr}$ ,  $16\mathrm{km/h}$ , 16 kilometres per hour). Certain quantities do not have units (e.g. When the relation is number of episodes).
4. Object entity is a date: Wikipedia uses only three date formats. We first find all dates in a sentence using regular expressions and parse them into a structured format containing day of the month, month and year. If any of these components exist in both the dates to be matched, they should match. For example, if the triple date has all three components but the date extracted from a sentence has only the year, then only the year needs to match.
5. Relations with a subproperty: For instance, the relation award received has the subproperty year and the relation spouse may have the subproperty place of marriage.
$^{2}$ https://en.wikipedia.org/wiki/Wikipedia:Date_formatting
Input Triples	Target Sentence
Das Tagebuch der Anne Frank, (distributor, Universal Pictures), (country, Germany), (publication date, 03 March 2016)	The film was theatrically released in the Germany on March 3, 2016, by Universal Pictures International.
Neff Maiava, (date of birth, 01 May 1924), (date of death, 21 April 2018), (occupation, professional wrestler)	Maiava (May 1, 1924 April 21, 2018) was an American Samoan professional wrestler.
Barack Obama 2012 presidential campaign, (country, United States), (end time, 06 November 2012), (start time, 04 April 2011)	The 2012 reelection campaign of Barack Obama, the 44th President of the United States, was formally announced on April 4, 2011.
Blue whale (parent taxon, Balaenoptera)	The blue whale (Balaenoptera musculus) is a marine mammal belonging to the baleen whale suborder Mysticeti.
We retain the main triple as such and reformat the subproperty as a triple of the form (subject-entity, object-entity subproperty_name, subproperty_value) e.g. (Barack, spouse, Michelle) has the subproperty (place of marriage, Trinity Church). These are converted to (Barack, spouse, Michelle) and (Barack, Michelle place of marriage, Trinity Church).
While the type of the triples is important in the alignment process, the verbalization model is ag- nostic to the type and treats all triples the same.
We perform a two-step sequential finetuning of the pre-trained T5-large (Raffel et al., 2020) model for converting triples to text. Triples are concatenated as subject relation_1 object_1, ....relation_n object_n for input to T5. The model is first fine-tuned on the aligned corpus for 5000 steps to increase the coverage of entities and relations. However, this results in the generation of Wikipedia-like sentences and hallucination if a certain expected input triple is missing. For example, Wikipedia sentences generally mention date of birth, date of death, occupation together. If the occupation is missing in the input, the system hallucinates a random occupation. "Neff Maiava date of birth 01 May 1924, date of death, 21 April 2018." generates "Neff Maiava (1 May 1924 - 21 April 2018) was an Albanian actor"; hallucinating a profession. To overcome this, we further fine-tune the model on WebNLG 2017 data for 500 steps. While WebNLG has low coverage, the information in the input triples matches the target sentence exactly. WebNLG also has a different sentence structure than Wikipedia. This reduces conformity to Wikipedia sentence structure and hence reduces hallucination. We use a learning rate of 0.001, a batch size of 1048576 tokens and a maximum decoding length of 256.
Table 2: Examples of alignment (training data).
Pearson correlation	0.73
Spearman correlation	0.66
Kendall's Tau	0.51
Table 3: Semantic Filtering Evaluation.
2.3 Quality Filtering
We perform a semantic quality based filtering of the sentences generated by the triple-to-text module. This is a separate post-processing module used during inference and is not jointly optimized with the text generation module. A semantic quality score is assigned to each generated sentence w.r.t. the input triples that indicates whether or not the generated text captures the full meaning of the triples and does not hallucinate extra information. The score is generated using a BERT base uncased model with input of the form [CLS] concatenated-triples [SEP] reference-or-generated-sentence. It is fine-tuned for 1000 steps on the WebNLG 2017 human assessment data. The data contains system predictions submitted to the shared task rated on a scale of 1-3 for semantics and fluency. We use the semantics score and scale it to 0-1. We also add gold references with a score of 1. This results in 2706 examples,  $90\%$  of which are used for finetuning and the remaining for evaluation. High correlations are obtained between the predicted scores and human scores on the evaluation split (Table 3).
3 KELM Corpus
In this section, we utilize the TEKGEN model and filtering mechanism to build a synthetic corpus that captures the KG in natural language format.
3.1 Entity Subgraph
Datasets such as WebNLG have instances with grouped triples that can be expressed as a fluent sentence. Such groups are not available for a large KG and using one triple at a time for inference would lead to hallucination as training uses multi-
Figure 4: Entity Subgraph Creation Algorithm using relation co-occurrence counts based on relation-sentence alignment in the training data. Each entity subgraph consists of a maximum of five triples, all with the same subject entity. The first triple is chosen at random. The second triple is chosen such that its relation has the highest co-occurrence count with the relation in the first triple and so on.
ple triples per example. Therefore, we develop a strategy to create entity subgraphs based on relation co-occurrence counts i.e. frequency of alignment of two relations to the same sentence in the training data. The algorithm is shown in Figure 4. It produces  $\sim 18\mathrm{M}$  entity subgraphs from  $\sim 45\mathrm{M}$  triples so the final corpus will have  $18\mathrm{M}$  generated sentences corresponding to each entity subgraph.
3.2 Generation
For each entity subgraph, we concatenate all its triples as before. We perform top 5 sampling with a temperature of 0.5. The bottom  $1\%$  of the generated sentences are filtered out based on the semantic score assigned using the model in  $\S 2.3$ .
Model	Finetuning data	Inference data	Semantics	Fluency
mean	stdev	mean	stdev
T5-only	WebNLG	Triple	4.12	1.02	4.16	1.02
T5-only	WebNLG	Subgraph	3.97	1.14	4.15	0.87
— TEKGEN —	Subgraph	4.36	0.87	4.60	0.58
Table 4: Human evaluation of the generated corpora, on a scale of 1-5, for semantics and fluency.
Generation quality of the KELM Corpus is evaluated using human ratings on a random sample of 200 entity subgraphs. Automatic metrics such as BLEU (Papineni et al., 2002) or BERTscore (Zhang et al., 2019) cannot be used due to the lack of gold references. Following prior work, the generated text is rated for two aspects—fluency and semantics, on a scale of 1-5, where 1 means not at all fluent/does not capture meaning at all and 5 means completely fluent/fully captures meaning with no hallucination. We have eight annotators total and each example is rated by two of them. All annotators are linguists, NLP researchers or NLP practitioners and volunteered for the evaluation. We do not use any crowd sourcing platform. For each instance, scores of the two annotators are averaged to get the final rating. The Pearson correlation between the two sets of ratings is 0.56 for semantics and 0.43 for fluency.
We compare TEKGEN to two baseline systems. For both baselines, we fine-tune a T5-large model only on WebNLG 2017 data but use different inference input. For one system, we use one triple at a time as input, resulting in 524 instances from the 200 entity subgraphs. For the second, we use the entity subgraphs as input, resulting in 200 instances. Scores are shown in Table 4. Entity subgraphs during inference do not improve the mean scores but reduce the standard deviation of the fluency. In comparison, TEKGEN with inference using entity subgraphs improve both the semantics and fluency of the generated text. Both the mean scores are higher and the standard deviations are lower. It paraphrases canonical names of relations in the KG to more natural expressions more often. Some examples of generation using the two systems are shown in Table 5. In the second example, the relation 'inception' is paraphrased to 'started' using WebNLG_finetuning+Triple_Inference and 'founded' using TEKGEN+Subgraph_Inference, the latter being more appropriate for organizations.
For completeness, we evaluate two more base
Input Triples	WebNLG_Finetuning + Triples_Inference	TEKGEN + Subgraph_Inference
(Michelle Obama, height, +71 inch)	Michelle Obama's height is +71 inch.	Michelle Obama is 71 inches tall.
(10x10 Photobooks, instance of, Nonprofit organization), (10x10 Photobooks inception, 00 2012)	The 10x10 Photobooks are the result of a non-profit organization.
10x10 Photobooks was started in 00 2012.	10x10 Photobooks, founded in 2012 is a nonprofit organization.
(Edu (footballer, born 1949), member of sports team, Tigres UANL)
(Edu (footballer, born 1949 ), Tigres UANL end time, 01 January 1983)
(Edu (footballer, born 1949 ), Tigres UANL start time, 01 January 1978)	Edu was born in 1949 and is a member of Tigres UANL.
Edu (footballer, born in 1949 ) Tigres UANL's end time was 01 January 1983.
Edu (footballer, born 1949 ) was at Tigres UANL from 01 January 1978.	Edu, who was born in 1949, played for Tigres UANL between 1978 and 1983.
(To Kill a Mockingbird, award received, Pulitzer Prize for Fiction)
(To Kill a Mockingbird Pulitzer Prize for Fiction point in time 00 1961)
(To Kill a Mockingbird Pulitzer Prize for Fiction winner Harper Lee)	To Kill a Mockingbird won the Pulitzer Prize for Fiction.
To Kill a Mockingbird was Pulitzer Prize for Fiction, awarded in 00 1961.
Harper Lee was the winner of the Pulitzer Prize for Fiction for To Kill a Mockingbird.	Harper Lee won the 1961 Pulitzer Prize for Fiction for To Kill a Mockingbird.
(Caucasus Mountains, country, Georgia (country))
(Caucasus Mountains, instance of, Mountain range)
(Caucasus Mountains, country, Russia)
(Caucasus Mountains, highest point, Mount Elbrus)
(Caucasus Mountains, country, Armenia)
(Caucasus Mountains, topic's main category, Category:Caucasus Mountains)	The Caucasus Mountains are located in Georgia.
The Caucasus Mountains is an example of a Mountain range.
Caucasus Mountains is in Russia.
The highest point in the Caucasus Mountain -s is Mount Elbrus.
Caucasus Mountains is in the country of Armenia.
The Caucasus Mountains is categorised as a Caucasus Mountains.	The Caucasus Mountains are a mountain range found in Georgia, Armenia and Russia. Mount Elbrus is the highest point in the Caucasus Mountains.
Table 5: Examples of text generated by the final model in comparison to the model trained only on WebNLG.
line systems in which T5-large model is finetuned only on the KG-Text aligned corpus but use the two different inference inputs—single triple and entity subgraphs. One annotator rated the same sample for semantics. The former had an average score of 2.34 and the latter 2.73. Since these scores were very low, we did not pursue the evaluation of these systems further. The use of just the aligned corpus which is noisy to some extent results in the worst performing system out of all the methods.
4 Knowledge Enhanced LMs
In this section, we showcase an application of the generated KELM Corpus as a way to integrate KGs into natural text corpora for pre-training language models (LMs), as shown in Figure 5. We choose REALM (Guu et al., 2020) as a representative of the recently introduced family of retrieval language models and therefore we expect our work to be equally applicable to other such language models. We show gains on LAMA knowledge probe and open domain QA with augmentation. We also perform experiments where we integrate raw Wikidata triples instead of KELM corpus to confirm the effectiveness of verbalization.
4.1 Retrieval Language Models
REALM is a retrieval-based language model and uses two corpora for pre-training—a retrieval corpus and a pre-training corpus. During pre-training, a sentence is selected at random from the pre-training corpus and a random word or salient span (dates and entities) is masked in this sentence. Then using a joint representation of the masked sentence and each of the documents in the retrieval corpus, the masked word is predicted. In the finetuning stage, the model is provided with a query/question as input in place of masked sentence from the pretraining corpora. It retrieves a small set of documents from the retrieval corpus based on the vector similarity of the query and document representation and then selects a span of text from the retrieved documents as the answer.
4.2 KELM Documents
We group sentences in the KELM corpus by subject entities to create 5722974 (5.7M) documents. We call these KELM documents. We then replace/augment the retrieval corpus in REALM with these synthetic documents. KELM Corpus has only  $\sim 286\mathrm{M}$  words ( $\sim 14\%$ ) in comparison to  $\sim 2\mathrm{B}$  words in English Wikipedia.
Figure 5: Knowledge Graph verbalization for integration with natural text corpora for language model pre-training.
We perform evaluation using two open domain question answering datasets and one knowledge probing dataset.
4.3.1 Question Answering
NaturalQuestions (NQ) (Kwiatkowski et al., 2019): Queries to Google and their answers.
WebQuestions (WQ) (Berant et al., 2010): question-answers from the Google Suggest API.
We keep the same settings as REALM for both NQ and WQ i.e. we work on the open domain setting for both datasets where no passage is provided as context for each question. Finetuning is performed on respective training splits.
4.3.2 Knowledge Probe
LAMA (Petroni et al., 2019): Fill-in-the-Blank style factual queries with single token answers from four sources: Google-RE,  $^3$  T-REx (Elsahar et al., 2018), SQuAD (Rajpurkar et al., 2016) and ConceptNet (Speer and Havasi, 2012). Google-RE also consists of aliases of each answer.
REALM did not include LAMA as one of its evaluation datasets. So we first evaluate REALM on LAMA using the original retrieval corpus and then using the KELM Corpus. No finetuning is performed and the masked word predictions from the pre-trained models are used as answers.
We evaluate REALM on WQ, NQ and LAMA under three settings by modifying the retrieval corpus.
1. ORIGINAL: Wikipedia text
2. REPLACED: only KELM Corpus
3. AUGMENTED: Wikipedia text + KELM Corpus