Hallucination Mitigation in Natural Language Generation from Large-Scale Open-Domain Knowledge Graphs
Authors: Xiao Shi
Zhengyuan Zhu
Zeyu Zhang
Chengkai Li
Venue: EMNLP 2023 Main (2023)
Keywords: graph-to-text generation
knowledge graphs
TL;DR: This paper proposes methods and datasets to the problem of natural language generation for large-scale, open-domain knowledge graphs.

Abstract: In generating natural language descriptions for knowledge graph triples, prior works used either small-scale, human-annotated datasets or datasets with limited variety of graph shapes, e.g., those having mostly star graphs. Graph-to-text models trained and evaluated on such datasets are largely not assessed for more realistic large-scale, open-domain settings. We introduce a new dataset, GraphNarrative, to fill this gap. Fine-tuning transformer-based pre-trained language models has achieved state-of-the-art performance among graph-to-text models. However, this method suffers from information hallucination---the generated text may contain fabricated facts not present in input graphs. We propose a novel approach that, given a graph-sentence pair in GraphNarrative, trims the sentence to eliminate portions that are not present in the corresponding graph, by utilizing the sentence's dependency parse tree. Our experiment results verify this approach using models trained on GraphNarrative and existing datasets. The dataset, source code, and trained models are released at https://github.com/idirlab/graphnarrator.

Hallucination Mitigation in Natural Language Generation from Large-Scale Open-Domain Knowledge Graphs
Xiao Shi, Zhengyuan Zhu, Zeyu Zhang, Chengkai Li
Department of Computer Science and Engineering The University of Texas at Arlington
Arlington, TX, USA (xiao. zhenguuanngunn..o z zeyu. zhang uta. edu, cli@uta.edl
Abstract
mining. The output is a textual sequence that de- scribes the fragment of triples.
generating natural language descriptions for knowledge graph triples, prior works used ei- ther small-scale, human-annotated datasets or datasets with limited variety of graph shapes, e.g., those having mostly star graphs. Graph- to-text models trained and evaluated on such datasets are largely not assessed for more re- alistic large-scale, open-domain settings. We introduce a new dataset, GraphNarrative, to fill this gap. Fine-tuning transformer-based pre- trained language models has achieved state-of- the-art performance among graph-to-text mod- els. However, this method suffers from infor- mation hallucination--the generated text may contain fabricated facts not present input graphs. We propose novel approach that, given graph-sentence pair in GraphNarrative, trims the sentence eliminate portions that are not present in the corresponding graph, by utilizing the sentence's dependency parse tree. Our experiment results verify this ap- proach using models trained on GraphNarrative and existing datasets. The dataset, source code, and trained models are released at https: //gi thub. com/ iidiridrrrr///eiieeee haaarraaaggaaaaaaa graphnarrator.
Verbalizing triples from knowledge graphs is crucial in variety of tasks and applications, in- cluding systems created for querying knowledge graphs (Liang et al., 2021; Jayaram et al., 2016) as well as systems backed by knowledge graphs for question-answering (Zhou and Small, 2019; Ma et al., 2018) and fact discovery (Xian et al., 2019; Zhang et al., 2018). In these places, knowledge graph fragments must be conveyed to users in var- ious forms, such query results and discovered facts. Though a tiny part of a whole knowledge graph, such graph fragments can still be complex and thus challenging to comprehend. Instead, pre- senting them in natural language can help end users understand them better.
In graph-to-text generation, the preciseness and naturalness of the textual narration of graph frag- ments is important. Generating high-quality text can be particularly challenging for large-scale and open-domain knowledge graphs. Specifi- cally, benchmark datasets in this line of research either are hand-crafted and monotonous, c.g., WebNLG (Gardent al., 2017a) only include simple, special formations narrated input frag- ments, e.g., EventNarrative (Colas et al., 2021) and TEKGEN (Agarwal et al., 2021). Existing graph- to-text models, being trained and evaluated on these datasets, are largely not validated for more realis- tic large-scale, open-domain settings. Section 2 presents this analysis in detail.
1 Introduction
The task of graph-to-text generation aims to auto- matically produce natural language descriptions of knowledge graphs. A knowledge graph G stores factual information as subject-predicate object triples, where each triple (8, p, o) corre- sponds to an edge from the subject entity s to the object entity o. The graph-to-text generation task entails, given a subgraph GCG, generating token sequence (J1..., yn) to describe G. This task can be accomplished by constructing machine learning models (Clive al., 2021; Castro Ferreira al., 2019; Trisedya al., 2018). The input such model is graph itself small fragment of triples from a knowledge graph, as the outcome of some upstream operation, e.g., search, query and data
This paper introduces GraphNarrative, a new dataset that fills the aforementioned gap be- tween graph-to-text models and real-world needs. GraphNarrative consists of around 8.7 million (input graph, output text) pairs. The text in each pair is Wikipedia sentence, whereas the corresponding graph comprises Freebase (Bollacker al., 2008) entities and relationships deseribed the sentence. The large-scale of both Wikipedia and Freebase, the

---

linguistic variation in Wikipedia, and the complex- ity of sentences and corresponding graph structures make this dataset more aligned with real-world scenarios. For instance, GraphNarrative'ss 's 8.7 mil- lion input graphs are in 7,920 distinct topological shapes and 22% of the 8.7 million are star graphs, in contrast to 94% and 96% in EventNarrative and TEKGEN, respectively. Section 3 articulates the details of GraphNarrative's 's creation.
We conducted comprehensive automatic and hu- man assessments of text descriptions generated by fine-tuned PLMs, specifically BART (Lewis et al., 2020) and T5 (Raffel et al., 2020). The automatic evaluation results consistently demonstrated that models performed better with the use of sentence trimming, across the datasets of GraphNarrative, TEKGEN, WebNLG, and DART (Nan et al., 2021). The approach led the increment of 12 and 7 points in BLEU score (Papineni et al., 2002) for GraphNarrative and TEKGEN, respectively. A T5- large model fine-tuned on GraphNarrative with sen- tence trimming achieved state-of-ther results on the WebNLG benchmark. Furthermore, human evaluation results showed that sentence trimming on average reduced 1. 1.4 entity hallucinations and relationship hallucination per text description.
Given the demonstrated efficacy of fine-tuning pre-trained language models (PLMs) in producing state-of-the-art results on graph-to-text (more de- tails Section 4), we adopt the same approach. As pointed out in (Agarwal et al., 2021; Dušek et al.. 2018), though, this approach may suffer from in- formation hallucination, i.c., the output texts may contain fabricated facts not present in input graphs. For example, given a two-triple input graph {(Nef Maiava, date of birth, 81 May 1924), (Neff Maiava, date death, 21 April 2018)), (Agarwal al., 2021) re- ported their model generates "Neff Maiava (1 May 1924- 21 April 2018) was an Albanian actor." Not only the input does not mention Maiava's profes- sion or citizenship, but also in the real-world he was an American Samoan wrestler instead.
The contributions of this paper are as follows.
A new dataset, GraphNarrative, that fills the gap between existing datasets and large-scale real- world settings.
The first to quantify hallucinations produced by graph-to-text models.
A novel approach, sentence trimming, to hal- lucination mitigation.
Comprehensive experiments and evaluations that verify the quality and utility of Narravvv,, as well as the effectiveness of sentence trimming.
Very few have considered how to mitigate hallucination in graph-to-text generation, except for (Agarwal et al., 2021; Wang et al., 2021; Ma et al., 2022). The first two studies attempted to ad- dress hallucination by further fine-tuning PLMs on WebNLG after fine-tuning on noisier automatically- extracted datasets. (Ma et al., 2022) adopted a dif- ferent approach, by filtering out training instances when the ROUGE-1 (Lin, 2004) scores between the input and the output fall below a certain thresh- old. However, these studies did not quantify the prevalence of hallucination in their models' outputs. Nor did they provide direct experiment results or other evidence to verify the approach in reducing hallucination. We are the first quantitatively measure the prevalence of hallucination in graph- to-text. We also developed a novel approach to mitigating hallucination by aiming at the problem's 's root moot-mismatch between graph and text in train- ing data. Given graph-text pair in GraphNarrative, the approach trims the text, i.c., Wikipedia sen- tence. by eliminating portions not represented in the graph. This process, named sentence trimming. is accomplished by analyzing the shortest paths between graph entities within the sentence's depen- dency parse tree (details in Section 5).
2 Limitations of Existing Datasets
First, most previous models were trained on small hand-crafted datasets that contain limited entity types and relations. For instance, WebNLG in- cludes 2,730 distinct entities and 354 distinct rela- tions. In contrast, real-world knowledge graphs can be much larger. For example, according to (Heist et al., 2020), Wikidata (Vrandečić and Krötzsch, 2014) has 52,252,549 entities, 2,356,259 classes, 6,236 relations, and 732,420,508 triples. The hand- crafted approach cannot scale to these massive knowledge graph, as it is impossible to manually write training graph-text pairs for so many different entity types, relations, and topic domains.
Second, the text descriptions in hand-crafted datasets such as WebNLG tend to follow monotonous templates, plausibly because the exam- ples were written by small number of human con- tributors. This limits the capability of trained mod- els to use diverse expressions narrating graph fragments. This lack of linguistic variation can hamper the usability of a text generation system.

---

Third, the graph fragments in existing datasets are largely limited simple star graphs (each graph consisting of a center entity and some of its one-hop neighbors) or more general acyclic graphs (i.e., one or more trees). The graphs in WebNLG have 41 distinct topological shapes (Appendix D), out of which 32 are acyelic graphs. The cycles are all 2-edge loops or self-loops. In DART, 83% of the graphs are star graphs. In automatically- generated datasets EventNurrative and TEKGEN, 94% and 96% of the graphs are star graphs, respec- dataset, AGENDA (Koncel-Ketee Kedziorski et al., 2019), has only 2% star graphs. But it only contains 7 dis- tinct relations in the special domain of scientific research. On the contrary, in practical scenarios the imput fragments can be of complex, general rather than simple, special formations. While di- rect measurement is lacking, we used the graphs described Wikipedia sentences as a proxy for gauging the shape diversity of graphs that need to be narrated. We manually analyzed the formations of graphs presented in 100 random Wikipedia sen- tences, and we found only 39 of the 100 graphs are star graphs. Similar but automatic analysis of the complete Wikipedia corpus (more details in Sec- tion Figure 2) show that only 2 of the 10 most frequent graph formations are star graphs, and are cyclic graphs.
Mergantown, West Virginia
place birth
John Dougles
Crucationnnstiuuee
Leaingron, Kertucky
Tetee Creek High School
Born in Morgentown, West Virghta, John Dougles graduated from Tates Creek High School in Lesington, Kerlucky.
Figure graph-sentence pair in GraphNarrative
Our customized entity linking solution consists of coreference resolution (McCarthy and Lehnert, 1995), wikification (Csomai and Mihalcea, 2008), and Wikipediaatoo-Freebbee
tively. Another automatically-colle
entity mapping. The entity mapping (more details in Section B.3) cre- ated 4, 4,408,115 one-to-one mappings between En- glish Wikipedia entities (i.e., articles) and Freebase entities, through a combination of three engineer- ing methods- using existing mapping in Free- base, by using Wikidata the midpoint connecting Wikipedia and Freebase entities, and similarly by using DBpedia (Auer et al., 2007) the midpoint. For wikification, our simple approach maps a span of tokens in a Wikipedia article D to a Wikipedia entity, the tokens exactly match either the entity? ss full title or any of the entity's wikilink anchor text in the same article D. For coreference resolution, we applied the implementation (Lee et al., 2017) in AllenNLP (Gardner et al., 2017) Wikipedia articles to replace pronouns and aliases with corre- sponding entities. The results of aforementioned processes were put together-a Wikipedia entity appearance in a Wikipedia sentence, either origi- nally as a wikilink or detected through wikification upon coreference resolution, Icads to the detection of the corresponding Freebase entity via the map- ping results.
3 The GraphNarrative Dataset
This section explains how we generated our new dataset GraphNarrative by aligning Wikipedia texts with Freebase. Note that the methodology could be applicable to text corpora beyond Wikipedia and knowledge graphs beyond Freebase. This section also contrasts GraphNarrative with existing bench- mark datasets to demonstrate how addresses cur- rent datasets' limitations.
Edge defection. Given the Freebase entities de- tected from Wikipedia sentence W, it identifies Freebase edges between the entities such that the corresponding relations are described in W. Given a pair of such entities, Freebase contains only one edge between them, our simple method assumes the corresponding relationship is deseribed in W. If Freebase has multiple edges between them, we include the edge whose label tokens overlap with W. If there are still multiple such edges, we in- clude the edge that is most frequent in Freebase. All these detected edges form the graph G that pairs with W an instance (G, W) in the dataset. Note that the simple assumptions in this approach may lead to both false positives and false negatives. In practice, the resulting dataset has solid quality
3.1 Dataset Creation: Graph-Text Alignment
For each applicable Wikipedia sentence W, we cre- ate the corresponding subgraph in Freebase, to form a graph-sentence pair (G, W) as one example instance in the dataset. See Figure for an example. This is achieved by an entity linking step followed by an edge detection step.
Entity linking. It maps span of tokens in the Wikipedia sentence W to an entity e in Freebase.
1Or 3 out of the 10, depending on whether considering a 3-node path as star or not.

---

Dataset
Knowlege Graph
Text Corpus
Domain
Instances
Entities
Triples
Relation
Star Craphs
WebNLG
DBpedia
Handcraft
15 DBpedia categories
25,298
2,730
3.221
354
57%
DART
NEA
Handeraft
N/A
38,391
27,000
32,139
3,834
83%
AGENDA
N/A
Scientific abstract
Scientific rescarch
40,720
159,691
177.568
2%
EventNamative
Wikidata
Wikipedia
Events
224,428
305,685
649,337
94%
TEKGEN
Wikidata
Wikipedia
Open domain
7,895.789
4,856,439
11,373,838
GraphNamative
Freehase
Wikipedin
Open domain
8,769,634
1,853.752
18,472,249
1,724
22%
Table 1: Comparison of graph-to-text datasets
(detailed assessment in Section 6.2). Nevertheless, workflow of dataset creation allows for more advanced and accurate methods in each component.
learns representations of input graphs and the de- coder subsequently translates the representations into token sequences. In decoder-only models, e.g.. GPT-2 (Radford et al., 2019), GPT-3 (Brown et al., 2020) and ChatGPT (i.e.. GPT-3.5) (Ope- nAI, 2022) in the GPT family, the decoder uses text sequence embedding as the representation of imput graphs. Encoder-decoder models fall into two main categories based on graph representations- models that encode lin- earized graphs' token sequences with LSTMs (Trisedya et al., 2018; Gardent al., 2017b) or Transformers (Castro Ferreira et al., 2019), and 2) models that use a dedicated graph encoder to capture the structural information of knowledge graphs (Schmitt et al., 2021; Ribeiro et al., 2020; Marcheggiani and Perez- crezz-Beltrachin,, 2018).
3.2 Characteristics of Graph
This section qualitatively and quantitatively ana- lyzes how GraphNarrative bridges the gap between graph-to-text models and real-world settings.
Scale and variety of entities and relations. GraphNarrative contains 8,769,634 graph-sentence pairs, 1,853,752 entities, 15,472,249 triples, and 1,724 relations from 84 Freebase domains (sce Ap- pendix B. As Table shows, most other datasets are significantly smaller in these aspects.
1)
Linguistic variation. Using Wikipedia as the corpus, the graph-text pairs GraphNarrative allow a model to learn from many Wikipedia authors' diverse narrations. On the contrary, text in hand- crafted datasets such as WebNLG and DART tend to follow monotonous templates from small num- ber of human contributors.
Many of the aforementioned models fine-tune PLMs. Fine-tuming transformer-bases based, sequence- to-sequence encoder-decoder PLMs (e.g.. T5 and BART) achieved state-of-the-e performance WebNLG (Ribeiro al., 2021; Wang al., 2021; Clive et al., 2021) and DART (Aghajanyan et al., 2021). (Yuan and Färber, 2023) reported that fine- tuning T5 and BART on WebNLG and AGENDA datasets yielded better results than zero-shot learn- ing using GPT-3 and GPT-3.5. They also reported factual hallucinations from GPT-3 and GPT-3.5. While the reported results from fine-tuning GPT- 2 on WebNLG (Harkous et al., 2020) worse than current state-of-the--ar, no results have been reported based on fine-tuning GPT-3 or GPT-3.5.
Graph structure complexity. The graphs in GraphNarrative contain 1-15 triples and 2-20 en- tities, in 7,920 distinct topological shapes based on graph isomorphism. (Detailed distributions of graph instances and shapes are in Appendix B.2.) Figure 2 displays the 10 most frequent shapes along with their instance counts. Furthermore, only 22% of the instance graphs are star graphs. On the con- trary, ExentNarrative and TEKGEN are dominated by star graphs, Table shows.
Following the lfte-of-the-an approach, we also fine-tuned T5 and BART on GraphNarrative and other datasets comparison. training and ap- plying a graph-to-text model, an instance graph is linearized into a token sequence. Following the method in (Ribeiro et al., 2021), the graph in Figure would be linearized as "<H> John Dou- glas <R> place of birth <T> Morgantown, West Vir- ginia <H> John Douglas <R> education institution <T> Tates Creek High School <H> Tates Creek High School
5,345,453
1,535,004
281 .000
222,021
188.611
191.230
150.750
82,100
60.674
$1
Figure 2: most frequent graph shapes GraphNarrative, with instance counts
4 Models
Existing graph-to-text models often use a decoder- only (Brown et al., 2020) or encoder-decoder struc- ture (Sutskever et al., 2014), where the encoder
<R> location <T> Lexington, Kentucky" where the special tokens <H>, <R> and <T> denote subjects, re- lations and objects, respectively.

---

5 Mitigation of Hallucination
source
punct
The culprit of the hallucination problem discussed in Section I is fabrication in training data-textual descriptions containing information not found in input graphs. This is evidenced by that, while graph-to-texi models frequently produce hallucina- tion when trained on TEKGEN, it rarely happens on WebNLG. Hallucinated facts are seldom found in the clean, manually-crafted WebNLG but are present in automatically extracted graph-text pairs in TEKGEN due extraction errors.
rondj
companed
puncl
loosely
advmod
Seftwan
Sadup
del
an
modeled
prep
based
after
for
grep
pabj
Theee
poss
Apple
Linux
Figure 3: Dependency parse tree of sentence "FlyBack is an open-source Backup Software for Linux based on Git and modeled loosely after Apple's 's Time Machine."
There could be two plausible directions in tack- ling graph-to-texl hallucination. One is to improve our graph-text alignment method (Section 3. The graph extracted from a piece of text during align- ment may miss certain entities or relationships due to either extraction errors disparities be- tween the text corpus and the knowledge graph. The resulting graph-text pair may misguide the trained model to hallucinate facts. A more accurate alignment method can reduce such erroneous pairs and thereby reduce hallucination. However, this method has an inherent limitation-sinee a knowl- edge graph in real-world is often far from complete, there will be facts in text that cannot be mapped the knowledge graph. Nevertheless, in princi- ple, way combine this approach with the other approach discussed below is open for investigation. This study explores a different direction in miti- gating hallucination. Given (Freebase subgraph G, Wikipedia sentence W) pair produced by align- ment, we introduce a sentence trimming algorithm (Algorithm in Appendix A) to turn W into trimmed sentence Wtrim by eliminating portions that are not present in G while preserving the sen- tence's main idea. Below we provide sketch of the algorithm, while keeping its pseudo code and description in Appendix A.
DPT of the sentence W in its caption. The cor- responding graph G from the graph-text align- ment process is {(FlyBack, software_genre, Backup toftware), (FlyBack, systatinng_ssstm,, Linux), (FlyBack, basis, Git)). Note that entities Apple and Time Machine in W are missing from G. The SDPs for the three triples are (L, ④), (0, @, @ b), and (Ø, @ C 6, )), respectively. Given the SDPs, min _oop is attained by FlyBack and mar_pe is attained by Git. Hence, Wtrim is "FlyBack is an open-source Backup Software for Linux based on Git". The sequence "and modeled loosely after Apple's Time Machine.", related the missing entities Apple and Time Machine, is trimmed from W.
Note that, a regular DPT will break up entities such as Backup Sof tware into individual tokens, each for node in the DPT. To avoid that, we used a modified concept of DPT-we preprocessed entity names and tokenized each entity's name into a sin- gle token. Speficially, the two tokens Backup and tware were combined into token BackupSof tware.
Results
6.1 Datasets
First, the algorithm parses W and generates its dependency parse tree (DPT) Wirees using spaCy (Honnibal et al., 2020). Then, for each triple 1, = (st, Pi, 0:) E G, identifies the short- est dependency path (SDP) between 8; and o i.e.. the shortest path between the two entities' tokens in Wtrce- It then finds the lefumost position in- dex min_pos in sentence W among all tokens on all triples' SDPs, and similarly the rightmost po- sition index max_pos. This process results in the trimmed sentence Wtrim. sub-sequence of W spanning from min_pos to max__os,
We performed experiments four datasets: GraphNarrative, TEKGEN (the large-scale, open- domain graph-to-text dataset that resembles ours the most), and WebNLG and DART (two human- annotated datasets). Detailed statistics about these and other datasets can be found in Table 1.
GraphNarrative is partitioned into training, de- velopment and test sets in accordance with the process elaborated below. Each edge in Free- base belongs a topic domain. Every instance in GraphNarrative, i.e., a (graph, sentence) pair, is assigned a domain, using the most frequent do- main among the graph's 's edges. We then divided
An example is in Figure which illustrates the

---

GraphNarrative into seen and unseen partitions ac- cording to the numbers of instance pairs in different domains. Domains with very few (less than 2,000) pairs were designated as unseen domains, while the remaining domains are seen. A full list of the seen and unseen domains is in Appendix B.I. All instances in unseen domains go to test set. In the seen partition, 90%, 5% and 5% of the instances are allocated for training, development and test, respec- tively. This resulted in 7,880,214 instances in the training set, 437,514 in the development set, and 451,906 in the test set, including 13,453 instances from unseen domains. Having unseen instances in the test set helps us evaluate models' generalization ability. Choosing domains with limited discussion ensures that the model has encountered only a few such instances during pre-training of PLMs.
ST
Hallncinated Entities
Missed Entities
Hallucinated Relations
Missed Relations
Grammar
w/o
1.163
0.003
1.340
0.040
4.793
w/
0.306
0.003
0.453
0.083
4.613
Table 2: Human evaluation of GraphNarrative quality
6.2 Human & Automatic Evaluation Metries
Human evaluation metrics. We evaluated the qual- ity of both the GraphNarrative dataset and the sen- tences generated by models, focusing on whether sentences in the dataset or produced by models fabricate facts that are not in the corresponding graphs narrated by the sentences. To the best of our knowledge, no prior study has quantitatively eval- uated the quality of graph-to-text datasets or mod- els with regard to hallucination. Specifically, we define the following four metrics: numbers of hal- lucinated entities (entities not present in the graph but mentioned in the sentence), missed entities (en- titics present in the graph but not mentioned the sentence), hallucinated relations (relations not present in the graph but mentioned in the sentence), and missed relations (relations present in the graph but not mentioned in the sentence).
In TEKGEN, each instance pair contains Wikipedia sentence and Wikidata subgraph ex- tracted from the sentence. We used the origi- nal training, development and test set partitions from (Agarwal et al., 2021). I). We could not use all instances due to lack of mappings between entity names and their surface texts. Without such infor- mation sentence trimming cannot be applied. To maximize the utility of available instances, we used aliases sourced from TEKGEN and leveraged regu- lar expressions to identify time and people's 's names. Consequently, we obtained 3,81 1,28 instances for training, 476,439 for development, and 484,958 for test, out of the original 6,310,061, 788,746, and 796.982 instances, respectively. . In the standard WebNLG 2017 challenge dataset, each instance is composed of graph from DBpedia and one multiple sentences written by human annotations to describe the graph's 's content, Its test set is divided into the seen partition, which contains DBpedia categories present in the train- ing and development sets, and the unseen partition, which covers 5 categories absent from the training and development sets. We used the same partition- ing in the dataset.
In addition, we also evaluated the quality of sen- tences using average grammar errors per sentence, on a scale of 1-5: 5 (no errors), (one error), 3 (two to three errors), 2 (four to five errors), and (more than five errors)
Automatic evaluation metrics. For model- generated sentences, we also report automatic evaluation results using standard natural lan- guage generation metrics BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005) and chrF++ (Popović, 2015).
6.3 Experiment and Evaluation Results
1) GraphNarrative dataset quality. Three hu- man annotators evaluated the quality of the graph- sentence pairs in GraphNarrative. We randomly chose 100 pairs, where each sentence has the origi- nal version and the trimmed version using the algo- rithm in Section The total 200 pairs were then shuffled so that annotators cannot tell whether a sentence is original or not. Each human annotator scored all 200 pairs using the metrics in Section 6.2, and their scores were averaged.
DART is data-to-text dataset that comprises pairs of (triple-set, sentence) gathered from vari- ety of sources, including WebNI.G, E2E (Novikova et al., 2017), and sentences collected through crowdsourcing and paired with tables extracted from WikiSQL (Zhong et al., 2017) and WikiTable- Questions (Pasupat and Liang, 2015). We used the original partitioning of training, development and test sets in DART.
Table 2 presents the results. In this and subse- quent tables, sentence trimming is denoted ST. The average hallucinated entities and relations per
WW used the metrics defined in Section 6.2 for evaluat- ing both dataset quality and model output. This may cause confusions in understanding the measures in the context of

---

Model
ST
BLEU
METEOR
chri+++
all
seen
unseen
all
seen
unseen
all
seen
unseen
BhRT-base
w/o
33.18
33.33
27.52
17.18
17.20
14.63
36.56
36.74
30.75
BART base
w/
46.49
46.77
36.67
24.43
24.53
21.30
49.92
50.12
43.29
BART
W/o
32.35
32,48
27.56
17.45
17.53
15.07
37.12
37.29
31.58
BART-lage
46.04
46.18
40.98
24.35
24:41
22.17
49.69
49.85
44.72
T5-small
w/o
19.48
19.53
17.34
15.78
15.85
13.79
33.92
34.08
28.90
T5-small
43.72
43.87
38.11
23,40
23.48
21.10
48.15
48.31
42.65
TS-base
W/o
16.89
16.95
14.63
16.23
16,30
14.10
35,37
35.54
29.84
base
42.18
42.29
37.85
24.27
21.94
49.63
49.80
Tallagg
W/o
22.22
12.26
20:41
17.16
17.23
15.02
36.78
36,95
31,40
TS-large
45.12
43.40
24.77
24.84
50.44
50.60
Table 3: Model performance on GraphNarrative
Model
WebNLG
et al., 2021), which adapted PLMs from Hugging Face (Wolf et al., 2019) for graph-to-text. The au- tomatic evaluation results of different models on GraphNarrative are in Table 3. Fine-tuning the T5- large model attained the best performance across most metrics, consistent with findings on WebNLG in (Ribeiro et al., 2021; Wang et al., 2021).
BLEU METEOR chrP++ BLEU METEOR chrF++
T5-large
4.01
9.54
24.64
3.44
7.93
23.17
Fitte-T5
19.81
30.36
54,01
16.15
27.53
48.69
GN-T5
21.38
31.82
56,83
19.35
27.35
50.41
GNST-T5
27.60
32.27
56.81
19.42
28.07
50.96
Table 4: Zero-shot performance of models on WebNLG and DART test sets
3) Graphtiraative in enhancing generalization ability. To assess if GraphNarrative may enhance PLMs' generalization ability, we conducted both zero-shot learning and fine-tuning experiments em- ploying GN-T5 and GNST-T5 on WebNLG and DART, where GNST-T5 denotes the fine-tuned T5- large model on GraphNarrative with sentence trim- ming, and GN-T5 denotes the counterpart without sentence trimming. They are also compared with the original T5-large model as point of reference. Zero-shot results. For zero-shot learning, we directly applicd the above-mentioned three models on the test sets of WebNLG and DART. The results are in Table 4. The results reveal that fine-tuning PLM on GraphNarrative substantially improves its generalization capabilities.
graph-sentence pair are 1.163 and 1.340, respec- tively. This reflects the challenges in graph-to-text alignment and the source of hallucination, ex- plained in Section 5. Applying sentence trimming reduced these numbers 0.306 entities and 0.453 relations, clearly showing its effectiveness in en- hancing graph-text alignment. On the other hand, when graphs were extracted from corresponding sentences to form GraphNarrative, information not present in the sentences was seldom introduced into the graphs, reflected in the small missed entities and relations, both less than 0.1. Sentence trim- ming only slightly increased missed relations from 0.040 to 0.083, showing insignificant side effect of removing from sentences information covered in corresponding extracted graphs. With regard to grammar, while sentence trimming led to slight decline in the grammar score, the difference (4.793 vs. 4.613) is not substantial.
Fine-tuning results. We subjected the three mod- els to further fine-tuning on WebNLG and DART for 100 epochs with an early stopping patience of 20 epochs, while keeping other hyperparameters consistent with those in Part 2, Section 6.3. No trimming
2) Model performance on GraphNarrative. We fine-tuned various T5 (small: 60M parame- ters, base: 220M parameters, and large: 770M parameters) and BART (base: 140M parameters, large: 400M parameters) models on GraphNarrative for 10t steps with a batch size of 8 using the Adam optimizer (Kingma and Ba, 2014) and an initial learning rate of 3 x 10-5, We employed a linearly decreasing learning rate schedule without warm-up and set the maximum target text length to 384 to- kens. Our implementation was based on (Ribeiro
performed on WebNLG and DART, as their sentences were authored by human annota- tors, with very few hallucinated or missed entities and relations. Table compares the performance of different graph-to-text models on WebNLG G test set, including the reprint of the results from seven prior studies. GNST-T5 fine-tuned on WebNLG outper- formed others on most metrics, particularly in the unseen category. This improvement suggests that GraphNarrative enhances the generalization ability of PLMs. Table 7 shows the fine-tuning results on DART test set. The model performance improve- ment by sentence trimming is not obvious. This is further discussed in Part 4, Section 6.3.
dataset quality-a hallucinated entity refers to an entity from the original sentence that is missed in the corresponding ex- tracted graph! We decided to tolerate this potential confusion for the sake of consistent metric definition.

---

Model
BLEU
METEOR
chr
seen
unseen
all
seen
unseen
seen
unseen
(Gardent al., 2017b)
31.24
$2.39
6.13
37.00
7.00
(Marcheggiani and Prrez-Belrachhni, 2018)
55.90
39,00
(Ferreira et al., 2019)
51.68
56.35
38.92
32,00
41.00
21.00
(Ribeiro et al., 2020)
63,69
44,47
76.66
(Ribeiro et al., 2021)
59.70
64.71
53.67
44.18
45.85
42.26
75,40
78.29
72.25
(Wang et al., 2021)
60.56
66.07
53.87
44.00
46.00
42.00
(Aghajanyan et al., 2021)
56.30
64.80
46.10
42,00
46,00
38.00
GNST-T5 (ours)
61.46
66.49
55.35
44.30
46.23
42.08
76.20
79.35
72.76
Table 5: Performance comparison of different graph-to-tex models on WebNLG test set
Dataset
ST
BLEU
METEOR
chrF++
all
seon
unseen
all
socn
unseen
all
scca
unseen
TEK
w/o
60.43
65.49 54.32
44.06
46.04
41.90
75.73
78.83
70.13
GEN
60.82
65.42
55.12
44.25
46.13
42.18
76.16
79.11
72.35
Graph w/o (GN-TS)
60.26
65.44
54.06
44.08
45,90
41.98
75.83
79.02
72.35
Narrative (GNST-TS) 61.46
66.49
55.35
44.30
46,23
42.08
76.20
79.35
72.76
Table 6: Models' performance on WebNLG test set, when fine-tuned with TEKGEN or GraphNarrative and further fine-tuned with WebNL.G
Model
BLEU
METEOR
chrF++
ble 6 also shows the results of (4). In these sults, the fine-tuned PLMs on GraphNarrative and TEKGEN with sentence trimming consistently out- performed their non-trimming counterparts. These findings underscore the effectiveness of sentence trimming in enhancing PLM performance. worth noting that, as Tables 6 and 7 show, on human-annotated WebNLG and DART the models did not gain much from sentence trimming after they are fine-tuned on these datasets. The main reason is that human-annotater annotateo datasets generally have well-aligned graph-text pairs and thus cannot be substantially improved by trimming.
TS-large
50.38
39.98
68.06
GN-T5
50.53
39.99
68.15
GNST-T5
50.51
40.07
68.23
Table 7: tine-tuning results on DART test set
is
Model
ST
BLEU
METEOR
chrF++
BARTlarge
w/o
41.31
23.62
47.13
BARTHarge
w/ 48.32
29.90
57.50
T5-large
w/o
43.03
24.21
48.05
Th-lare
w/
49.83
30.52
58.25
Table 8: Performance of fine-tuning BART-large and T5- large on the TEKGEN dataset
4) Ablation study of sentence trimming. We demonstrate the effectiveness of sentence trimming in improving model performance on GraphNarrative, TEKGEN, WebNLG, and DART by fine-tuning PLMs with and without sentence trimming, respec- tively. (I) For GraphNarrative, we fine-tuned T5 and BART models using the setup described in Part 2, Section 6.3. (2) For TEKGEN, we fine-tuned the Ta-lerge and BART-large models using the serial- ized triples from (Agarwal al., 2021), with the same hyperparameters as in Part 2, Section 6.3. (3) For WebNLG and DART, we conducted zero-shot learning and fine-tuning experiments as described in Part 3, Section 6.3. (4) Additionally, on the WebNL. dataset, we carried out further fine-tuning of the T5-large model fine-tunedd on TEKGEN in (2), applying the same hyperparameters in Part 3. Section 6.3.
5) Sentence trimming in mitigating halluci- nation. We randomly sampled 100 graphs from GraphNarrative test set, along with the correspond- ing sentences generated by GNST-T5 and GN-T5. We shuffled the 200 pairs and used three human evaluators score the pairs, in the same fashion as in Part 1, Section 6.3. The results are in Table 9, which shows a reduction of 1.4 hallucinated enti- ties and 1.0 hallucinated relations per instance from GN-T5 to GNST-T5, suggesting that sentence trim- ming effectively mitigates hallucinations. Further- more, sentences generated by both models exhibit on average less than 0.07 missed entities and 0.38 missed relations per instance. Regarding gram- mar, sentences generated by GNST-TS received
The results of (1) and (2) are in Tables 3 and 8. The metrics (BLEU, METEOR, chrl++) consis- tently improve with sentence trimming, further verifying the efficacy of sentence trimming. The results of (3) are in Tables 4, 6 and 7, and Ta-
ST Entities
Hallucinated
Missed Entities
Hallucinated
Missed Relations
Grammar
Relations
wio
1.643
0.063
1.363
0.240
4.613
w/
0.260
0.056
0.300
0.370
4.356
Table 9: Human evaluation of sentences generated by T5-large with and without sentence trimming

---

Input Graph
Sentence Generated by GN-T5
Sentence Generated by GNST-TS
[Arthur Morry, place of death, Brishane)
Arthur Morry diad in Brishane. Queenslond, aged
Arthur Morry died in Brishane. US Naval Academy Annapolis, Maryland.
[Anrapolis Maryland, land, location Academy)
Naval
US Noval Academy Annapolis, Maryland was renamed the US Naval Academy Annapolis, Maryland, and the US Naval Acadenyy Amsapolis, Maryland Academy Annapolis, Maryland. Goldie Gers Along is 1981 American comedy him directed Malcolm Clair (6lmmaker) starring Lili and Chartes Morton (actor).
Daring World War
was renamed US
(Golii Ciets Along, him directed by, Malcolm St. filmmaker) mance actor, Lili Damita (Geldie Gets Along, film performance actor, Charles
Goldie Gets Along was directed Malcolm and starred Lili Damita and Charles Morton (actor).
Along. perfor-
(filmmaker)
actor)
Table 10: Comparison generated sentences with and without sentence trimming for sample imput graphs
slightly lower scores than GN-T5. Nevertheless, these scores remain acceptable, with on average less than one grammar error per instance.
instances and tested on non-star instances. These results indicate that a PLM fine-tuned on a dataset consisting solely of star graphs performs poorly when applied to general graph shapes, which are commonly encountered in real-world applications. Fine-tuning PLMs diverse graph shapes en- hances their generalization capability.
Table 10 illustrates the sentences generated by GNST-T5 and GN-T5 for few input graphs. GN- T5 tends to fabricate facts that are incorrect or non- existent in the real world (e.g., Arthur Morry's age of death, the renaming of the US Naval Academy, and Goldie Gets Along's year of release) not present in input graphs (e.g., Goldie Gets Along's genre). In contrast, GNST-T5 generated fluent sentences without fabricating facts, barring a phrase instead complete sentence for the second example.
Star
Non-star
train 290,047
1,022,303
1,312,350
dev 16.192
56,612
72,804
16,425
58,517
74,942
all
322,664
1,137,432
1,460,096
Table 1I: Number of star and non-star instances in GraphNarrative
6) L Limitations of star graph datasets. As plained in Section 1, existing large-scale datasets such as TEKGEN contain predominantly star graphs. We used GraphNarrative to investigate the limitations star graph datasets.
ST
Train
Test
BLUE
METEOR
chrl+++
36.57
22.75
46,70
30.64
21.99
45.48
both
33.54
22.32
46.09
More specifically, we separated the graph- sentence pairs in GraphNarrative into star instances (with star graphs) and non-star instances (without star graphs). We excluded instances with two or three entities, as they could be considered as both paths and stars. Table 11 provides the distributions of these two types of instances. The number of non- star instances in all three sets is approximatcly 3.5 times many as the star instances. To help ensure a fair comparison, we randomly selected an equal number of non-star instances as the star instances for each of the three sets, e.g., there are 290,047 star graphs and the same number of non-star graphs in the training set of our prepared dataset.
star
34.02
21.67
44.54
w/o
non-star
non-star
37.18
23.99
50.02
both 35.71
22.99
47.57
36.23
22,94
47.06
both non-star
36.86
24.32
5U.61
both
36.56
23.63
48,83
47.70
26.62
$2.08
star non-star
37.75
25.17
50,20
both
42.48
25.88
51.14
star
45.83
25.72
50.33
w/
non-star 47.30
27.73
55.21
both
46.61
26.73
$2.77
star 47.60
26.87
52.52
both
non-star
46.83
27.89
55.45
both
47.20
27.38
53.99
Table 12: Model performance, star vs. non star graphs
Using the dataset prepared this way, we fine- tuned T5-large model with and without sentence trimming for 10 epochs under early stopping pa- tience 5, using the same other hyperparameters as in Part 2, Section 6.3. The results are in Ta- ble 12. Across the board, models trained using star instances exhibited the highest performance when tested using star instances too, and similarly regarding non-star instances. Furthermore, models trained on non-star instances and tested on star in- stances tended to outperform models trained on star
7 Conclusion
In this paper, we proposed a novel approach to mitigating hallucination in natural language gen- eration from large-scale, open-domain knowledge graphs. We released large graph-to-tex dataset with diverse graph shapes that fills the gap between existing datasets and real-world settings. The ex- periment results show the effectiveness of our hal- lucination mitigation approach as well as the use- fulness of the dataset.

---

Limitations
scriptions of knowledge graphs, we are particularly aware of the potential misuse of our method for the generation of false, deceptive, biased or unfair con- tents. Particularly, our sentence trimming method aims to minimize such potential misuse by aiding in reducing hallucinations.
1) The creation of Graphtarrative and the sentence trimming method leverage an existing mapping be- tween the knowledge graph entities and Wikipedia entities. Given other text corpora and knowledge graphs, creating such mapping is non-trivial undertaking that often requires named entity recog- nition and disambiguation techniques. 2) The sen- tence trimming approach may introduce grammati- cal errors into generated sentences. 3) The method focuses on describing the content of an input graph only, without considering context information such neighboring entities in the knowledge graph. Such extra information may be preferred by user given certain application contexts may make the input graph's 's narration more natural. 4) The creation of GraphNarrative does not consider mul- tiary relationships in knowledge graphs. More specifically, the Freebase used in our work a version in which multiary relationships were con- verted into binary relationships Shirvani-Mahhan et al., 2023). In general, there is a lack of inquiry into multiary relationships in graph-to-text models. To the best of our knowledge, the only work in this area that discusses such multiary relationships is (Agarwal et al., 2021) and they also converted multiary relationships into binary ones. 5) A cou- ple of studies (Agarwal et al., 2021; Wang et al., 2021) attempted to address hallucination by further fine-tuning PLMs on WebNLG after fine-tuning on noisier automatically exxtracted datasets. It will be informative to conduct a human evaluation com- parison between their approaches and the sentence trimming method proposed in our work. Similarly, our future work includes a human evaluation com- parison with the filtering-bait based method (Ma et al., 2022) which we empirically compared with in Ap- pendix C.I. 6) The sentence trimming algorithm only removes irrelevant portions from the begin- ning and the end of a sentence, leaving the token sequence in the middle intact. It is possible the middle portion also contains tokens irrelevant to the input graph.
We also recognize that natural language descrip- tions generated using dataset and algorithm can be repurposed in various ways. We firmly urge users and developers use this content re- sponsibly, particularly with respect to intellectual property rights. Furthermore, we recommend users clearly label Al-generated content, promoting trans- parency and trust.
Data privacy and bias. Our GraphNarrative dataset uses publicly available data, particularly Freebase and Wikipedia, which do not contain in- formation that violates anyone's privacy to the best of our knowledge.
Our reliance on Wikipedia may inadvertently introduce bias, as Wikipedia content can reflect the views of its contributors. We are also aware this potential bias could be more intense in less commonly spoken languages, where the number of contributors might be limited.
A Acknowledgments
This work is supported by the National Science Foundation under Grants 1719054, 1937143, and 2333834. We extend our gratitude to Dr. Won Hwa Kim and Xin Ma for provisioning of computational resources in supporting this work. Special thanks to Nasim Shirvani- their vital contributions to the human evaluation process, to Yogesh Gurjar for resolving bug our experiment scripts, and to Mohammed Samiul Saeef for his guidance and expertise on using Free- base data graph.
and Haiqi Zhang for
References
Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami A1-Rfou. 2021. Knowledge graph based synthetic corpus generation for knowledge-enhanced language model pre- traininng..
Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 3565.
Ethics Statement
In the course of conducting research, have striven remain aware and attentive potential ethical implications and challenges. Our work was informed by the following ethical considerations.
Armen Aghajanyan, Dmytro Okhonko, Mike Lewis. Mandar Joshi, Hu Xu, Gargi Ghosh, and Luke Zettlemoyer. 2021. HTLM: Hyper-text pre-training and prompting of language models. arXiv preprint arXiv:2107.06955.
Ethical use of generated content. Given that our research focuses on producing natural language de-

---

Giuseppe Attardi. 2019, Wikiextractor. https:// github. com/oto///e//ooeee htt////ttrriiiiiierrrttooo. Accessed:
in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing, pages 552-562.
Sep 20th, 2019.
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. DBpedia: A nucleus for web of open data. In Proceedings of the 6th International Semantic Web Conference and 2nd Asian Semantic Web Conference, pages 722-735.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez- Be-ltractthii. 201 Creating train- ing corpora for NLG micro-planning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 179-188.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltracce Beltrachini. 2017b. The WebNLG challenge: Generating text from RDF data. In Pro- ceedings of the 10th International Conference on Natural Language Generation, pages 124-133.
Satanjeev Banerjec and Alon Lavie. 2005, Meteor: An automatic metric for mt evaluation with improved cor- relation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summariza- tio, pages 65-72.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2017. AllenNLP: deep semantic natural lan- guage processing platform. In arXiv preprint arXiv: 07660.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: A col- laboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, pages 1247-1250.
Hamza Harkous, Isabel Groves, and Amir Saffari. 2020. Have your text and use it too! end-to-end neural data-to-text generation with semantic fidelity. In Pro- ceedings of the 28th International Conference on Computational Linguistics, pages 2410-2424.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Nicolas Heist, Sven Hertling, Daniel Ringler, and Heiko Paulheim. 2020. Knowledge graphs the web-av overview. Knowledge Graphs for Explainable Ar- rificial Intelligence: Foundations, Applications and Challenges, pages 3-22.
Thiago Castro Ferreira, Chris van der Lee, Emiel van Miltenburg, and Emiel Krahmer. 2019. Neural data- to-text generation: A comparison between pipeline and end-to-end architectures.
Proceedings of the 2019 Conference on Empirical Methods in Natural I Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 552-562.
Matthew Honnibal, Ines Montani, Sofie Van Lan- deghem, and Adriane Boyd. 2020. spaCy: Industrial- strength natural language processing in Python.
Nandish Jayaram, Rohit Bhoopalam, Chengkai Li, and Vassilis Athitsos. 2016. Orion: Ennhling sugges- tions in a visual query builder for heterhhetroogeeouus graphs. arXiv preprint 16X5v:6605.0680
Jordan Clive, Kris Cao, and Marek Rei. 2021. Con- trol prefixes for text generation. arXiv preprint 11x.v.2110.08329.
Anthony Colas, Ali Sadeghian, Yue Wang, and Daisy Zhe Wang. 2021. EventNarrative: large- scale event-centric dataset for knowledge graph-to- text generation. In Proceedings of the 35th Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Jan- Christoph Klie. 2022. wikimapper. https:// // gi wwtp///www//w/w//eecccceeooo
Andras Csomai and Rada Mihalcca. 2008. Linking doc- uments to encyclopedic knowledge. IEEE Intelligent Systems, 23.
Rik Koncel-Kedziorski, Dhanush Bekal, Yi Luan, Mirella Lapata, and Hannaneh Hajishirzi. 2019. Text generation from knowledge graphs with graph trans- formers. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, pages 2284-2293.
Ondfej Dušek, Jekaterina Novikova, and Verena Rieser. 2018. Findings of the E2E NLG challenge. In Pro- ceedings of the 1Ith International Conference on Nauural Language Generation, pages 322-328.
Thiago Castro Ferreira, Chris van der Lee, Emiel Van Miltenhurg, and Emiel Krahmer. 2019. Neu- ral data-to-text generation: A comparison between pipeline and end-to-end architectures. Proceed- ings of the 2019 Conference on Empirical Methods
Kenton Lee, Luheng He. Mike Lewis, and Luke Zettle- moyer. 2017. End-to-end neural coreference reso- lution. In Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing. pages 188-197.

---

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising vequence-to-sequeme pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Länguistics, pages 7871-7880.
Panupong Pasupat and Percy Liang. 2015. Composi- tional semantic parsing on semi- structured tables. In Proceedings of the 53rd Annual Meeting of the As- sociation for Computational Linguistics and the 7th International Joinl Conference on Natural Language Processing (Volume 1: Long Papers), pages 1470- 1480. Association for Computational Linguisties.
Maja Popović. 2015, chrF:
character n- n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392-395, Lisbon, Portugal, Association for Computational Linguistics.
Shiqi Liang. Kurt Stockinger, Tarcisio Mendes de Farias, Maria Aniiimova, and Manuel Gil. 2021. Querying knowledge graphs in natural language. Journal of Big Data, 8(1):1233.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81. Association for Compu- tational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. 2019. Language models are unsupervised multitask learners.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Chao Ma, Chunhua Shen, Anthony Dick, Qi Wu, Peng Wang, Anton van den Hengel, and Ian Reid. 2018. Visual question answering with -uugenee networks. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 6975-6984.
Leonardo FR Ribeiro, Martin Schmitt, Hinrich Schütze, and Iryna Gurevych. 2021. Investigating pretrained language models for graph-to-text generation. In Pro- ceedings of the 3rd Workshop Natural Language Processing for Conversational AI, pages 211-221
Kaixin Ma, Hao Cheng. Xiaodong Liu, Eric Nyberg, and Jianfeng Gao. 2022. Open domain question answering with a unified knowledge interface. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 1: Long Papers), pages 1605-1620.
Leonardo FR Ribeiro, Yue Zhang, Claire Gardent, and Iryna Gurevych. 2020. Modeling global and local node contexts for text generation from knowledge graphs. Transactions of the Association for Compu- tational Linguistics, 8:589-604.
Diego Marcheggiani and Laura
2018. Deep graph convolutional encoders for structured data to text generation. In Proceedings of the 11th International Conference on Natural Language Gen- eration, pages 1-9.
Martin Schmitt, Leonardo FR Ribeiro, Philipp Dufter, Iryna Gurevych, and Hinrich Schütze. 2021. Mod- eling graph structure via relative position for text generation from knowledge graphs. In Proceedings of the Fifteenth Workshop on Braphh-Based Methods for Natural Language Processing, pages 10-21.
Joseph F McCarthy and Wendy G Lehnert. 1995. Us- ing decision trees for coreference resolution. arXiv preprint cmp- hmp-1g/9505043.
Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad. Chiachun Hsieh, Xian- gru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, et al. 2021. DART: Open-domain structured data record to text generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 432 447.
Nasim
Farahnaz Akrami, Mo- hammed Samiul Sacef, Xiao Shi, and Chengkai Li. 2023. Comprehensive analysis of freebase and dataset creation for robust evaluation of knowledge graph link prediction models. In Internationaa Se- mantic Web Conference. Springer.
Jekaterina Novikova, Ondrcj Dušck, and Verena Rieser. 2017. The e2e dataset: New challenges for end-to- end generation. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 201 -06..
Ilya Sutskever, Oriol Vinyals, and Quoe V Le. 2014. Sequence to sequence learning with neural networks. Advances in Neural Information Processing Systems, 27.
Bayu Distiawan Trisedya, Jianzhong Qi, Rui Zhang, and Wei Wang. 2018. GTR-LSTM: A triple encoder for sentence generation from RDF data. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 1627-1637.
OpenAl. 2022. httpss///ppee wwwpppenan..oeeeeeeeeee chatttt..t..
ChatGPT [large language model].
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. BLEU: a method for automatic eval- uation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguistics, pages 31 1-1-18..
Denny Vrandečié and Markus Krötzsch. 2014. Wiki- data: a free collaborative knowledge base. Communi- cations of the ACM, (70))888--

---

Qingyun Wang, Semih Yavuz, Xi Victoria Lin, Heng Ji, and Nazneen Rajani. 2021. Stage-wise fine- tuning for graph-to-text generation. In Proceedings of the ACL-IJCNLP 2021 Student Research Work- shop, pages 16-22.
A The Sentence Trimming Algorithm
In Algorithm 1, Lines 1-6 are to ensure that an en- tity consisting of multiple tokens is tokenized into one single token, and the mapping M is for recover- ing the entity's s tokens from W in producing Wtrim (Lines 15-16). Lines 9-14 find the leftmost posi- tion min_pos and the rightmost position max__pos from W by scanning each triple (s, p, o) in G and finding the tokens on the corresponding SDPs, as explained Section The variable node denotes a token on the SDP in Wtree between entities s and o. node.start and node.end denote node's starting position index and ending position index in W, respectively. node.start, node.end, min pos, and maz maz_pos
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. 2019, Huggingface's transformers: State-of- the-art natural language processing, arXiv preprint urXiv:1910.03771.
Yikun Xian. Zuohui Fu, Shan Muthukrishnan, Gerard De Melo, and Yongfeng Zhang. 2019. Reinforcement knowledge graph reasoning for explainable recom- mendation. In Proceedings of the 42nd International ACM SIGIR Conference on Research Develop- ment in Information Retrieval, pages 285-294.
character level. If node ap- pears multiple times in W, node.start will be the start index of the first one and node.end will be the end index of the last one.
Shuzhou Yuan and Michael Färber. 2023. Evaluating generative models for graph-to-text generation. arXiv preprint arXiv:2307. 14.
Gensheng Zhang, Damian Jimenez, and Chengkai Li. 2018. Maverick:
B More Details about GraphNarrative
Discovering exceptional facts from knowledge graphs. In Proceedings of the 2018 ACM SIGMOD International Conference Management of Data, pages 1317-1332.
B.1 List of Domains
Seen domains. location, people, sports, music. government, organization, education, film, tv, book, award, military, soccer, time, geography, business, olympics, transportation, broadcast, baseball, fic- _una__uiverrs,, biology, influence, language, com- puter, cvg, aviation, architecture, protected _itess, religion, symbols, travel, visual_ basketball, royalty, astronomy, american football, metropoli- tan_ uin_trnnst, somic_books law, _adma__ spaceflight, tennis, boats, medicine, meteorology, automotive, theater, internet, amusement_ parks, event, comic_sttiii measurement_ _unit, finance, radio, physics
Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning, arXiv prepriat Xxx:1709900010
Zhou and Kevin Small. 2019. Multi-domain dia- logue state tracking as dynamic knowledge graph enhanced question answering. arXiv preprint arXiv:1911.06192.
Algorithm Sentence Trimming
Input: W: A co-reference resolved Wikipedia sentence; G: The graph for W based on graph-text alignment
Output: W Wtrim: The trimmed text sequence
Unseen domains. martial arts, games, ice ice_hockss hockcy, cricket, rail, food, opera, projects, dining, skiing, conferences, library, exhibitions, zoos, boxing, en- gineering, digicams, venture venture_capital, chemistry, celebrities, chess, interests, distilled_spiife com- edy, fashion, geology, wine, bicycles
M +
foreach (s, o) E do
+ s.remooeesspcciaa_foe [ooeess)
reeeove(sppccial tokens)
W& + repplace(((s, o), (5',
Mo'] +
Wore dedependency_puree
min_pos, mar_pos W.length, foreach (s,p, E
B.2 GraphNarrative Characteristics
10 sdp <- shortest path(Wirra Free, 8 o')
Graph distribution. The distributions of graphs in GraphNarrative by numbers of triples and entities are shown in base-10 10 logarithmic scale in Figure 4 and Figure 5, respectively. Furthermore, the dis- tribution of distinct GraphNarrative graph shapes by number of entities is in Table 13.
11
foreach node E adp do
12
min_pos +- min(minnnpon pos, start)
13
mar +- max(max__ppppaaa _ao_ node.end)
Werim W[min mosinppos:: max max__os] pos]
15 foreach E M.keys() do
L Werim Werim-replace(k: (kk)
17 return Wixim
Text distribution. The dataset exhibits an average sentence length of 34.68 tokens for original sen-

---

#entitics 2
5
8
#shapes
705
1690
1705
1267
830
Log (Instance count)
#entities 12
13
14
15
16
17
19
20
all
#shapes 542
378
222
176
106
58
52
22
12
7920
Table 13:
Distribution of distinct GraphNarrative graph shapes by number of entities
#Tokens
Original Sentence
Trimmed Sentence
0-10
64,861 (0.74%)
1,734,759 (19.78%)
2
7
9
10
11
12
-3
14
10-20
1,442,582 (16.45%)
3,369,747 (38.43%)
Number triples In graph
20-30
2,592,651 (29)56%%
1,944.225 (22.17%)
Figure 4: Distribution of GraphNarrative instances by num-
30-40
2,048.174 (23.36%)
919,554 (10.49%)
her triples graphs
40-50
1,215,928 (13.87%)
416,187 (4.75%)
50-60
649,403 (7.41%)
190.363 (2.17%)
60-70
339,671 (3.87%)
91,289 (1.04%)
70-80
181,699 (2.07%)
45,38 (0.52%)
80-90
97,618 (1.11%)
23,823 (0.27%)
Log (Instance count)
90-100
53,981 (0.62%)
13,299 (0. 15%)
100-110
83,066 (0.95%)
21,008 (0.24%)
Table 14: Distribution of GraphNarrative instances by
sentence length
ITrippess
#Tokens
#Triples
#Tokens
#Triples
#Tokknss
14.15
2
20.32
23.12
2
7
9
10
14 15 16 19
20
26.29
5
28.21
29.74
Number of entities in graph
31.12
33.41
30.39
Figure 5: Distribution of GraphNarrative instances by num- her of entities in graphs
35,90
39.53
42.49
43.60
49.71
50.21
Table 15: Average GraphNarrative sentence length by number of triples in graphs
tences and 20.66 tokens for trimmed sentences. Ta- ble 14 provides detailed distribution of sentence lengths. Table 15 presents the average sentence token counts by number of triples in the graphs. underscores that our model was trained using di- verse set of examples, including those with lengthy sentences and a substantial number of triples.
verse pairs. If the input graph triples to a graph- to-text model containing such reverse edges, we only need to simply retain one edge out of each redundant pair. Hence, we did exactly that in pre- processing the whole Freebase dump that our in- put graphs have reverse edges. Furthermore, pre-processing also removed the mediator (CVT) nodes (Bollacker et al., 2008) by concatenating edges connected through mediator nodes.
B.3
Dataset Creation
B.3.1
Pre-processing of text corpus and knowledge graph
Our text corpus is derived from the Wikipedia dump& 3 released on Sep. 1st, 2019. In preprocess- ing the corpus, we used WikiExtractor (Attardi, 2019) transform the compressed XML file into numerous plain text files containing bodies of Wikipedia articles without ta- bles, infoboxes, table of contents, categories, and so on.
B.3.2 Graph-text alignment
Mnkipedia-to-Freebble entity mapping. We col- lected Wikipediiatto-Freebass entity mapping between 4,408, English Wikipedia titles and their corresponding Freebase entities. The map- ping was created by employing three meth- ods, as follows. 1) Parsing the Freebase data dump to obtain a Wikipediaato--Frebbsse entity mapping using https: /tti///ittttbbbb com/saleiro/
Wikipedia dump in
Our Freebase knowledge graph is from (Shirvani- Mahdavi et al., 2023) which used the most recent Freebase dump as the data source. Most rela- tions the graph form semamtically-rredndaan re-
2) Inferring from Wikipediaato-Wikidaa mapping in wikimap- per (Klie. 2022) and Wikidataato-FFreebase mapping at https://developeet gwwgle..og.ooooooooooo.oooo.o... com/ freebase. 3) Inferring from the Wikipedia-to- DBpedia and the -opeedia-to-Frebaar mappings at http:///ownloads dbpedia.
htttps:/ddume wikimedia. thttps///en.n wikipedia. wttp://ppo.....rr.kk///kkk.aaaa org//ikk//ikkipeda:: Lanual_of_Style/LLayorr
hhttps://develoge google.com/freebae. com/freebase

---

Threshold
Dor (wlo
(T)
(wil ST)
(wio ST)
(wr (T/
ST)
Tobal (w) ST)
0.8
(0.016993)
791 (0. 18069))
65 (00014488)
931 (0.20066)
1328 (0.016586)
14787 (0.18775)
1467 (0.01785)
16509 ro. 188588))8)
0.5
14952 (3.4668)
71565 357%)
5845 (3, 907%))
75445 (16.689%)
269541 (3 (3.4188))
1293864 (16.40855)
300336 (3,4753)
1362074 (5.7)
(32.
256357
(32.5028))
s
1578921 (32.5489)
(86.)
Table 16: Number of remaining instances after filtering using different thresholds of ROUGE-1 similarity scores
Dataset
Rutities
Triples
Redations
Graphs
Shapes
sentences better with input graphs by trimming away portions of sentences, the filtering method removes graph-text pairs from the DART dataset where the ROUGE-I similarity score between the graph and the text below 0.8.
Filtered
307,
457,519
974
47%
597
1,8537522
15,472,249
1,724
22%
7,920
Table 17: Statistics of GraphNarrative and its filtered dataset
core-i18n/en/. The overall Wikipedia-to- Freebase entity mapping is obtained by com- bining all three methods and eliminating con- flicting entity mappings. The mapping file link can be found at https::/github.. http:://gtthubb.cmm//d//rrr graphnarrator.
We applied the same filtering method on GraphNarrative. Table 16 provides breakdown of the remaining instances after filtering using differ- ent thresholds. A relatively low threshold of 0.3 removed 43.32% of the instances in GraphNarrative. When we raised the threshold to 0.8, almost all in- stances were eliminated. In comparison, the thresh- old of 0.8 applied on DART allowed for retaining 88% of its instances. This is because the human- annotated DART has well-aligned graph-text pairs. We compared sentence trimming with filtering using the 269,541 instances left in the training set and 7 71,565 in the development set, under thresh- old 0.5. We fine-tuned the T5-large model for 10 epochs with early stopping patience 5, using the same other hyperparameters as on the full dataset. The number of training steps is different from the full dataset because this subset is about 30 times smaller. We used early stopping to avoid overfit- ting. Then we used the resulting model, which we call Filter-T5, for zero-shot prediction on WebNLG and DART test sets. The results are shown in Ta- ble 4. GNST-T5 slightly outperformed Filter-TS, To understand this, we compared the statistics of the filtered dataset and the full dataset (and thus the dataset after sentence trimming since trimming does not alter the graphs in the dataset), as in Ta- ble 17. The filtered dataset exhibits a significant reduction in size and diversity in terms of number of distinct entities, relations, triples and shapes. We conjecture that this contributes to its performance degeneration in comparison with GNST-T5.
Coreference resolution. To produce more graph- text pairs for GraphNarrative, we used AllenNLP's coreference resolution (Gardner al., 2017; Lee et al., 2017) in default settings to replace Wikipedia token spans with the entities they refer to. We con- ducted human evaluation to assess the quality of the coreference resolution results on 20 randomly se- lected Wikipedia articles. The assessment yielded precision of 91.4% (630 of the 689 resolved entity coreferences were correct) and a recall of 98.3% (11 entity coreferences were missed).
#Triples
BLEU (GN-T5)
BLEU (GNST-T5)
13.60
25.97
19,90
27.53
24.07
29.73
30.51
32.62
32,95
35.22
38.81
39.73
7
42.74
41.55
8
42.23
42.10
55.73
51.83
45.84
48.08
41.72
42.71
38,09
39.33
41.77
43.27
14
33.47
36.95
15
34.04
38.44
Table Distribution of GNST-T5 and GN-T5 model
performance in BLEU scores GraphNarrative test set
C.2 Performance of GNST-T5 and GN-T5 by input size
C Additional Experiments
C.1 Comparing sentence trimming with filtering
Table 18 shows the performance of GNST-T5 and GN-T5 in BLEU scores on graphs of varying sizes, i.e., number of triples. The results help gauge whether the models generalize well for long in- puts. Notably, the performance of both models on extended inputs is better than or on par with their performance on shorter inputs.
We compared sentence trimming with similar but different filtering method proposed in (Ma et al., 2022). Their method also aimed to reduce dispari- ties in datasets as a way of mitigating hallucination. However, different from our approach which aligns

---

D The 41 Shapes in WebNLG
Figure displays the distinct graph shapes in the WebNLG dataset, in descending order by number of instances.
1320
1131
(3)
[16)
[37]
(19)
(191
(25)
1450
(30)
1311
(12)
CX31
(35)
(39)
(90)
Figure 6: WebNLG shapes


Reviews:
[{"review_id": "KOTptrHgMQ", "reviewer": "Reviewer_TcQj", "summary": "", "questions": "1. Can you provide some evaluation result with large scale models like GPT-3.5. I wonder if the hallucination could be ealisy mitigated with models with large scale parameters, and the proposed sentence triming method is only suitable for small models as used in this paper.\n\n2. Will the transformer structures (encoder-only, encoder-decoder, decoder-only) affects the result ?\n\n3. Can you provide some explanations that why sentence triming seems not so effective on WebNLG 2017 (Table 7) ?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the thoughtful comments which will help us improve the camera-ready version of the paper. Please see below for our response to the comments.  \n\n \n\n**1.** “Only T-5-X and BART is evaluated on the new dataset, while the result for most recent large language models are not presented. Besides, the evaluation only conduct on LM with encoder or encoder-decoder based transformer structures, no result for decoder-only transformers models (e.g. GPT-X) is presented.” \n\n \n\nand  \n\n \n\n“Can you provide some evaluation result with large scale models like GPT-3.5. I wonder if the hallucination could be ealisy mitigated with models with large scale parameters, and the proposed sentence triming method is only suitable for small models as used in this paper.”  \n\n \n\n**Response**\n\n \n\nThank you for the comment. We identified a couple of studies that use GPT-X for graph-to-text tasks. (Harkous et al., 2020) used GPT-2, and a July 2023 pre-print (Yuan and Färber, 2023) used GPT-3 and chatGPT (GPT-3.5) on WebNLG and AGENDA datasets. Results from encoder-decoder models (T5, BART) still outperform the results of the decoder-only models reported in these publications. Furthermore, (Yuan and Färber, 2023) also shows that GPT-3 and GPT-3.5 still hallucinate facts. Nevertheless, we will discuss these publications and we would like to reproduce the results of using GPT-3.5, for inclusion in the camera-ready version of the paper.   \n\n \n\n[Harkous et al., 2020] Hamza Harkous, Isabel Groves, and Amir Saffari. 2020. Have Your Text and Use It Too! End-to-End Neural Data-to-Text Generation with Semantic Fidelity. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2410–2424, Barcelona, Spain.  \n\n \n\n[Yuan and Färber, 2023] Shuzhou Yuan, Michael Färber. \"Evaluating Generative Models for Graph-to-Text Generation.\" arXiv preprint arXiv:2307.14712 (2023). \n\n \n\n \n\n**2.** “Experimental settings for previous dataset is not clear. The author may make a more detailed introduction to the settings of the experiments on previous datasets like TEKGEN and WebNLG 2017. For example, how the dataset is splited and is the experimental setting remains the same with baselines.” \n\n \n\n**Response**\n\n \n\nWe used the same split (train, dev, test) provided by the TEKGEN and WebNLG. For WebNLG, we also used their split between seen and unseen domains. Furthermore, the training strategy on TEKGEN and WebNLG are the same as in (Ribeiro et al., 2021) which used T5-large model. We will make these details clear in the final version. Details about experiment settings for linearization of the graphs and other hyperparameters are mentioned in Lines 543-546 and Lines 559-572 of the paper.  \n\n \n\n[Ribeiro et al., 2021] Leonardo FR Ribeiro, Martin Schmitt, Hinrich Schütze, and Iryna Gurevych. 2021. Investigating pretrained language models for graph-to-text generation. In Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI, pages 211–227. \n\n \n\n \n\n**3.** “Sentence triming, which is the main method proposed by the author, seems not generalize well to all datasets. It shows a large performance gain on TEKGEN in Table 8. However, the performance gain seems pretty incremental in WebNLG as in Table 7.”   \n\n \n\nand  \n\n \n\n “Can you provide some explanations that why sentence triming seems not so effective on WebNLG 2017 (Table 7) ?” \n\n\t \n\n**Response**\n\n \n\nSentence trimming does not benefit human-annotated datasets (e.g., WebNLG, DART) much after fine-tuning on it. The main reason is that human-annotated datasets generally have good quality with graph and text well-aligned, i.e., the text exactly describes the content of the graph without extra information and very few hallucinations in their training data. Thus, they do not really need to be trimmed to remove parts that appear in text but not in the graph. On the contrary, in automatically collected datasets such as TEKGEN and GraphNarrative, the text more often contains more information than the graph, especially because real-world knowledge graphs are incomplete. For such kind of data, trimming will be more useful. We will provide more detailed analysis like this in the final version.  \n\n \n\nWe added the zero-shot experiments on WebNLG, results are shown in the below table. First fine-tuning on GraphNarrative improves the model’s generalization ability on WebNLG for zero-shot learning.  \n\n \n\n| Model    | BLEU  | METEOR | ChrF++ |\n| -------- | ----- | ------ | ------ |\n| T5-large | 4.01  | 9.54   | 24.64  |\n| GN-T5    | 21.38 | 31.82  | 56.83  |\n| GNST-T5  | 27.60 | 32.27  | 56.81  |\n\n \n\n \n\n**4.** “Will the transformer structures (encoder-only, encoder-decoder, decoder-only) affects the result?” \n\n \n\n**Response**\n\n \n\nWe have a hypothesis that the transformer structures may not affect the results, because sentence trimming doesn’t operate on the model. It only modifies the dataset. We’d like to experiment and verify this and report corresponding results in the final version. We are not sure which encoder-only model can be used for generation tasks. Does the reviewer have any suggestions?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Propose a ground new large scale dataset that solves the problems raised by previous datasets. The dataset contains various substructures and a large amount of relations which is aligned with real world knowledge graphs.\n\n2. To eliminate the hallucination problem, the author propose a novel method, sentence triming, to eliminate portions that are not present in the corresponding graph, by utilizing the sentence’s dependency parse tree, which is interesting.\n\n3. Extensive experiments are conducted through machine and human evaluations and the result almost aligned with the claimed contributions.", "reasons_to_reject": "1. Only T-5-X and BART is evaluated on the new dataset, while the result for most recent large language models are not presented. Besides, the evaluation only conduct on LM with encoder or encoder-decoder based transformer structures, no result for decoder-only transformers models (e.g. GPT-X) is presented.\n\n2. Experimental settings for previous dataset is not clear. The author may make a more detailed introduction to the settings of the experiments on previous datasets like TEKGEN and WebNLG 2017. For example, how the dataset is splited and is the experimental setting remains the same with baselines.\n\n3. Sentence triming, which is the main method proposed by the author, seems not generalize well to all datasets. It shows a large performance gain on TEKGEN in Table  8. However, the performance gain seems pretty incremental in WebNLG as in Table 7.", "questions_to_authors": "1. Can you provide some evaluation result with large scale models like GPT-3.5. I wonder if the hallucination could be ealisy mitigated with models with large scale parameters, and the proposed sentence triming method is only suitable for small models as used in this paper.\n\n2. Will the transformer structures (encoder-only, encoder-decoder, decoder-only) affects the result ?\n\n3. Can you provide some explanations that why sentence triming seems not so effective on WebNLG 2017 (Table 7) ?", "paper_topic_and_main_contributions": "This paper studies the problem of graph to text generation, which is widely used in search and knowledge graph query. In this paper, the author propose a new graph to text dataset which aligns with real world knowledge graphs with more complex substructurs and relation types than previous methods. Besides, to mitigation hallucination when generating, the author propose sentence triming to eliminate irrelevant contents in the generated sentence. Experimental results show the effectiveness of the proposed sentence triming methods and the quality of the datasets. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None at the moment", "Typos_Grammar_Style_and_Presentation_Improvements": "None at the moment", "Justification_for_Ethical_Concerns": "None at the moment"}, "extra_text": {}}, {"review_id": "FTsLOoaxXC", "reviewer": "Reviewer_RttK", "summary": "", "questions": "Regarding the proposed hallucination mitigation method, could a sentence after trimming be semantically incomplete (due to the lack of the trimmed sub-sentences)?\n\nIs there any possibility that the proposed sentence trimming method could falsely remove sub-sentence that aligns with the given graph (False Negatives)? \n\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for the thoughtful review which will help us improve the final version of the paper.  \n\n \n\n**1.** “The proposed approach requires a mapping between the graph entities and Wikipedia entities. If the mapping does not exist, the approach cannot be applied.” \n\n \n\n**Response** \n\nThis is a valid concern. When such a mapping is missing, techniques for NER, information extraction, and so on can be potentially applied to create such a mapping. The impact of such an algorithmically discovered mapping on the efficacy of graph-to-text models remains to be examined.   \n\n \n\n**2.** “Regarding the proposed hallucination mitigation method, could a sentence after trimming be semantically incomplete (due to the lack of the trimmed sub-sentences)?  \n\n \n\n“Is there any possibility that the proposed sentence trimming method could falsely remove sub-sentence that aligns with the given graph (False Negatives)?” \n\n \n\n**Response**  \n\nIf our interpretation of these questions is incorrect, please let us know and we’d love to engage in further discussions. Based on our interpretation, the two questions are about the possibility of sentence trimming leading to the removal of sentence subsequences that are actually depicted in the graph. In theory there could be two corresponding scenarios:  \n\n \n\na) Entities in the graph being trimmed from the sentence: this scenario is implausible. The trimming method consistently retains entities from the graph in the sentence, as they always reside on their shortest dependency path. \n\n \n\nb) Relationships within the graph being trimmed from the sentence: This scenario is possible, although the loss is likely outweighed by the gain. Our human evaluation results (Table 3 in the paper) show a marginal increase in missed relationships, from 0.04 to 0.083, but a much more significant reduction of fabricated relationships from 1.340 to 0.453. (There was a typo in the paper---in Table 3, 0.04 and 0.083 should be swapped. We apologize for this and we will fix it in the final version.)\n\n**3.** \"line 031-033: does the sentence mean \"The graph-to-text generation task entails the generation of a token sequence Y=(y1, ..., yn) given a subgraph to describe G\"?\"\n\n\n**Response**\n\nYes. Thank you! We will fix the grammar error.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper stands as a pioneering research on evaluating and mitigating the information hallucination issue inherent in graph-to-text models with PLMs. Besides, the author pointed out the limitations of existing datasets, i.e., they are small hand-crafted and contain limited entity types and relations. Besides, the text descriptions in the instances of hand-crafted datasets tend to follow monotonous templates. Also, the graph fragments in existing datasets are largely limited to simple star graphs or acyclic graphs. The author argues that models trained on such data may not be as effective on large-scale and open-domain knowledge graphs in the real world. In comparison, the proposed GraphNarrative dataset contains more than 8 million graph-sentence pairs and use Wikipedia as text corpus, which has diverse narrations of various relations. GraphNarrative also contains a total of 7,920 distinct topological shapes based on graph isomorphism.\n\nMoreover, the logical framework of the entire paper is clearly structured, the theoretical concept and experimental detail are thorough. The writing of this paper is good and easy to follow. Besides, the selected datasets are representative and the experiment settings are reasonable.\n", "reasons_to_reject": "The proposed approach requires a mapping between the graph entities and Wikipedia entities. If the mapping does not exist, the approach cannot be applied.", "questions_to_authors": "Regarding the proposed hallucination mitigation method, could a sentence after trimming be semantically incomplete (due to the lack of the trimmed sub-sentences)?\n\nIs there any possibility that the proposed sentence trimming method could falsely remove sub-sentence that aligns with the given graph (False Negatives)? \n\n", "paper_topic_and_main_contributions": "The academic paper introduced a new graph-to-text dataset named “GraphNarrative”, which offers a significant improvement over existing datasets by incorporating more realistic large-scale and open-domain settings. The paper further delves into a notable issue that is found within the approaches which applied pretrained language models to solve graph-to-text task – the phenomenon of information hallucination. Remarkably, this study is the first of its kind to quantitatively measure the prevalence of hallucination induced by graph-to-text models. To address this prevalent issue, the authors propose a  approach, which involves trimming the sentence according to its dependency parse tree and eliminates portions that are not represented in the graph in order to effectively eliminate hallucination. The paper reports thorough experiments and evaluations to demonstrate the utility and effectiveness of the newly introduced GraphNarrative and the novel approach proposed for combating hallucination.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 031-033: does the sentence mean \"The graph-to-text generation task entails the generation of a token sequence Y=(y1, ..., yn) given a subgraph to describe G\"?"}, "extra_text": {}}, {"review_id": "J1UQXoDEYf", "reviewer": "Reviewer_hPGF", "summary": "", "questions": "What is the average length of the sentences in your dataset? What is the sentence length distribution? If the sentences are mostly short, then the models trained on this dataset might not generalize well to tasks with long inputs.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the valuable review and suggestion which have helped us conduct more experiments that will be included in the camera-ready version of the paper. Thank you!  \n\n\n**1.** “The generalization of the proposed dataset is not well studied. For example, GraphNarrative is more than 300x larger than WebNLG dataset, and it covers a lot more graph types, entities, relations and domains than WebNLG. Intuitively, models trained on GraphNarrative should be able to generalize well to WebNLG without further finetuning. However, no zero-shot results are presented.” \n\n**Response**\n\nThank you so much for the comment, based on which we conducted three zero-shot experiments on the WebNLG dataset: 1) using original graph-to-text task T5-large model (Ribeiro et al., 2021) to directly perform inference, without fine-tunning on WebNLG; 2) T5-large fine-tuned on GraphNarrative without sentence trimming (GN-T5); and 3)T5-large fine-tuned on GraphNarrative with sentence trimming (GNST-T5). The three experiments used the same linearization method for the graphs and parameters mentioned in the submission.  The results are shown below, and we will discuss these results in the camera-ready version.  \n\n[Ribeiro et al., 2021] Leonardo FR Ribeiro, Martin Schmitt, Hinrich Schütze, and Iryna Gurevych. 2021. Investigating pretrained language models for graph-to-text generation. In Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI, pages 211–227. \n \n| Model    | BLEU  | METEOR | ChrF++ |\n| -------- | ----- | ------ | ------ |\n| T5-large | 4.01  | 9.54   | 24.64  |\n| GN-T5    | 21.38 | 31.82  | 56.83  |\n| GNST-T5  | 27.60 | 32.27  | 56.81  |\n\nAs the results show, the zero-shot performance of GN-T5 exceeds the original T5-large model by a large margin, indicating the GraphNarrative dataset improves the generalization ability of PLMs. GNST-T5 further improves the model’s performance, reflecting the effect of sentence trimming. We will also include results for more models (e.g., BART-based, GPT-based) in the camera-ready version.  \n\n \n\nThe zero-shot results of GN-T5 and GNST-T5 shown in the table are worse than the results from T5 models fine-tuned using  WebNLG (Table 7 in the submission). GNST-T5 and GN-T5 were not fine-tuned using WebNLG, and thus it is expected for a model fine-tuned using WebNLG to perform better when both models are compared on their inference accuracy measured by WebNLG. The reasons could be 1) the underlying knowledge graphs are different---WebNLG was generated using DBPedia as the underlying knowledge graph while GraphNarrative was generated using Freebase. 2) WebNLG is human-annotated, with relatively good alignment between graphs and texts, while GraphNarrative was automatically collected and thus has more misalignments. 3) the differences in evaluation scores could also be due to the more diverse expressions in Wikipedia sentences (in GraphNarrative) compared to those crafted by human annotators (in WebNLG). Automatic evaluation scores such as BLEU rely on n-gram similarity between predicted and ground-truth sentences. Consider a single-triple graph. A human annotator might use one particular expression, while the model generates an alternative expression using different wording. In such cases, even if the model's expression is accurate, the evaluation score could be considerably low.  \n\n \n**2.** “The necessity of applying the sentence trimming method is not well established. For example, in Table 3 and Table 4, the results show that training on the trimmed sentences leads to more grammatical errors in the outputs, suggesting that trimming could lead to unnatural sentences. What if you simply discard the sentences that do not align well with the graph? Given the large size of GraphNarrative, there should be a decent amount of instances left after pruning. The authors should also consider experimenting with different amounts of training data to examine the effect of training size.” \n\n \n\n**Response** \n\n \n\n**2.1** “What if you simply discard the sentences that do not align well with the graph? Given the large \tsize of GraphNarrative, there should be a decent amount of instances left after pruning.” \n\n \n\nTwo categories of misalignment can be discerned: entity misalignment and relationship misalignment. \n\nIn the context of entity misalignment, instances where certain entities within a graph fail to be referenced in the corresponding sentence have been deliberately omitted from the GraphNarrative dataset. It's important to note that it's quite common for sentences to encompass more entities than those present in the graph, given the inherent incompleteness of real-world knowledge graphs. Implementing such exclusion criteria would likely result in the removal of a substantial portion of instances. \n\n \n\nRegarding relationship misalignment, the challenge lies in accurately determining whether the relationships depicted in the graph are appropriately aligned with the sentence. The diversity of natural language expressions compounds this challenge, making it difficult to establish precise alignment. \n\nWe extend our gratitude for your valuable suggestion to utilize the filter method as described in (Ma et al. 2022). We have diligently applied this methodology to our dataset and will elaborate on our findings in our subsequent response. \n\n \n\n \n\n**2.2** “The necessity of applying the sentence trimming method is not well established. For example, in Table 3 and Table 4, the results show that training on the trimmed sentences leads to more grammatical errors in the outputs, suggesting that trimming could lead to unnatural sentences. \n\n \n\nIt's important to note that, while grammar errors slightly increase after applying sentence trimming, the reduction in hallucinations is much more significant. As Table 4 shows, with sentence trimming, numbers of hallucinated entities and hallucinated relations were reduced by 1.4 and 1.0, respectively, while grammar score only decreased from 4.613 to 4.356 (a score of 5 means no grammatical error, 4 means one error, and so on). This trade-off is especially relevant in applications where truthfulness and semantic coherence hold greater importance than grammar. Furthermore, we are actively engaged in designing methods to mitigate grammar errors. \n\n \n\n**2.3** \"The authors should also consider experimenting with different amounts of training data to examine the effect of training size.” \n\n \n\nThank you for the suggestion. We will report the results of such experiments in the camera-ready version.  \n\n \n\n**3.** “The comparison with recent related studies is lacking: for example, Nan et al. 21 proposed the DART dataset, which is used for studying data-to-text generation. In fact, DART could be another evaluation set for understanding the generalization of GraphNarrative. (Ma et al. 2022) proposed a data-to-text generation model. They also proposed to filter out instances from training data that might lead to hallucinations or missing information. The authors should consider discussing and comparing with these related works.” \n\n \n\n**Response** \n\n \n\nWe will include the discussion of the two papers in the camera-ready version. Based on the comment, \twe obtained statistics of the DART dataset and did additional experiments of zero-shot learning and fine-tuning the original T5 large, GN-T5, and GNST-T5 on DART. We also applied the filtering method by (Ma et al., 2022). Below we summarize the results, and we will include details in the paper.  \n\n \n\n[Ma et al., 2022] Kaixin Ma, Hao Cheng, Xiaodong Liu, Eric Nyberg, and Jianfeng Gao. 2022. Open Domain Question Answering with A Unified Knowledge Interface. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1605–1620, Dublin, Ireland.  \n\n \n\n \n\n**3.1** The distinctions between DART and the GraphNarrative dataset are as follows: \n\n \n\nGraph characteristics: A notable disparity is observed in the graph structures. Specifically, a significant majority (83%) of the graphs in DART take the form of star graphs. An overview of its shape distribution can be found in the table below. \n\n \n\nDataset nature: DART is primarily a human-annotated dataset. It integrates data from various sources including Wiki tables (human-annotated descriptions), WebNLG (also human-annotated), and E2E (human annotated too). In contrast, GraphNarrative is an automatically collected dataset. \n\n \n\nApplication focus: DART predominantly is for table-to-text generation. Notably, it lacks a dedicated, large-scale, open-domain knowledge graph akin to Wikidata, DBpedia, Freebase, and similar resources. This aspect diverges from our specific aim: the automated generation of natural language descriptions for subgraphs sourced from a vast, open-domain knowledge graph. \n\n \n\n \n\n|         | **Instances** | **Star Instance** | **[TABLECONTEXT] instance** | **Source counter**                                                                                                                                      | \n| ------- | ------------- | ----------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | \n| Dev     | 2768          | 2484              | 61                          | {'e2e': 1475, 'webnlg': 872, 'WikiSQL_decl_sents': 249, 'WikiTableQuestions_lily': 106, 'WikiTableQuestions_mturk': 39, 'WikiSQL_lily': 27}             | \n| Test    | 5097          | 4308              | 69                          | {'webnlg': 2753, 'e2e': 1844, 'WikiSQL_decl_sents': 346, 'WikiTableQuestions_lily': 95, 'WikiTableQuestions_mturk': 34, 'WikiSQL_lily': 25}             | \n| Train   | 30526         | 24976             | 3176                        | {'e2e': 12530, 'webnlg': 6940, 'WikiTableQuestions_lily': 4700, 'WikiSQL_decl_sents': 3589, 'WikiTableQuestions_mturk': 2047, 'WikiSQL_lily': 720}      | \n| **Sum** | **38391**     | **31768**         | **3306**                    | **{'e2e': 15849, 'webnlg': 10565, 'WikiSQL_decl_sents': 4084, 'WikiTableQuestions_lily': 4901, 'WikiTableQuestions_mturk': 2120, 'WikiSQL_lily': 772}** | \n\n \n\n \n\n**3.2** Tables below show the model performance of zero-shot learning and fine-tuning on DART. For fine-tuning, we trained for 100 epochs with early-stopping patience 20, and we picked the best performance model on development set, with linearization and other hyperparameters set the same as in . \n\n \n\nZero-shot results: \n\n \n\n| Model    | BLEU  | METEOR | ChrF++ | \n| -------- | ----- | ------ | ------ | \n| T5-large | 3.44  | 7.93   | 23.17  | \n| GN-T5    | 19.35 | 27.35  | 50.41  | \n| GNST-T5  | 19.42 | 28.07  | 50.96  | \n\n \n\nFine-tuning results: \n\n \n\n| Model    | BLEU  | METEOR | ChrF++ | \n| -------- | ----- | ------ | ------ | \n| T5-large | 50.38 | 39.98  | 68.06  | \n| GN-T5    | 50.53 | 39.99  | 68.15  | \n| GNST-T5  | 50.51 | 40.07  | 68.23  | \n\n \n\nThe zero-shot results show that fine-tuning GraphNarrative improves the models’ generalization ability by a large margin.  \n\n \n\nWhen fine-tuning DART, GN-T5 and GNST-T5 does not improve over T5-large much, similar to the results on WebNLG (Table 7 of submission). This is because both WebNLG and DART are human-annotated datasets without many hallucinations in their training dataset. As mentioned in Lines 176-180 of the paper, the human-annotated dataset is not realistic for a large-scale, open-domain setting. That is why we need the sentence trimming method, of which the effectiveness was demonstrated on both TEKGEN and GraphNarrative datasets. \n\n \n\n**3.3** We applied the filter method in (Ma et al. 2022) by filtering out instances in GraphNarrative dataset whose ROUGE-1 scores between graph and text fall below a threshold. We used three different thresholds: 0.8 (used by (Ma et al., 2022), based on communication with the authors of (Ma et al. 2022), 0.5, and 0.3. The numbers and percentages of instances above various threasholds are shown below.  \n\n \n\n| Threshold | Dev (w/o trimming)| Dev (w/ trimming) | Test (w/o trimming) | Test (w/ trimming) | Train (w/o trimming) | Train (w/ trimming) | Total (w/o trimming) | Total (w/ trimming) | \n|-----------|-------------------|-------------------|---------------------|--------------------|----------------------|---------------------|----------------------|---------------------| \n| 0.8       | 74 (0.0169%)      | 791 (0.1806%)     | 65 (0.0144%)        | 931 (0.206%)       | 1328 (0.0168%)       | 14787 (0.1877%)     | 1467 (0.017%)        | 16509 (0.1883%)     | \n| 0.5       | 14952 (3.416%)    | 71565 (15.357%)   | 15843 (3.507%)      | 75445 (16.689%)    | 269541 (3.418%)      | 1293864 (16.405%)   | 300336 (3.43%)       | 1382074 (15.76%)    | \n| 0.3       | 142974 (32.678%)  | 256357 (58.593%)  | 148666 (32.892%)    | 266094 (58.886%)   | 2579921 (32.740%)    | 4621448 (58.659%)   | 4970561 (56.68%)     | 7445899 (84.88%)    | \n\n \n\nHalf of the instances in our dataset were filtered out even with a low threhold of 0.3. Given 0.8 as the threshold, 99% of the instances were filtered out. The filtering method is not quite applicable to our dataset. DART essentially is a human-annotated dataset, and thus it is more clean than automatically collected dataset. After filtering out instances below score 0.8, there are still a large proportion of the instances (88%) remaining. A real-world knowledge graph is usually far from complete, as there could be many misalignmenst between the graph and text. By sentence trimming, we improve the ROUGE-1 score between graph and text, as can be seen by comparing the columns with and without sentence trimming in the table above.  \n\n \n\n \n\n \n\n**4.** “What is the average length of the sentences in your dataset? What is the sentence length distribution? If the sentences are mostly short, then the models trained on this dataset might not generalize well to tasks with long inputs.” \n\n \n\n**Response**\n\n \n\nThe dataset exhibits an average sentence length of 34.68 tokens for original sentences and 20.66 tokens for trimmed sentences. Additionally, the distribution of sentence lengths within our dataset is provided in the table below for reference. The majority of our dataset falls in the number of tokens from 10-40. Comparatively, DART boasts an average sentence length of 21.10 tokens, while WebNLG registers an average of 22.50 tokens. (Colas, Anthony, et al., 2023) Notably, the sentences within our dataset are not much shorter than those found in other comparable datasets.  \n\n \n\n| #tokens | original sentence  | trimmed sentence   | \n| ------- | ------------------ | ------------------ | \n| 0-10    | 64.861k (0.74%)    | 1734.759k (19.78%) | \n| 10-20   | 1442.582k (16.45%) | 3369.747k (38.43%) | \n| 20-30   | 2592.651k (29.56%) | 1944.225k (22.17%) | \n| 30-40   | 2048.174k (23.36%) | 919.554k (10.49%)  | \n| 40-50   | 1215.928k (13.87%) | 416.187k (4.75%)   | \n| 50-60   | 649.403k (7.41%)   | 190.363k (2.17%)   | \n| 60-70   | 339.671k (3.87%)   | 91.289k (1.04%)    | \n| 70-80   | 181.699k (2.07%)   | 45.38k (0.52%)     | \n| 80-90   | 97.618k (1.11%)    | 23.823k (0.27%)    | \n| 90-100  | 53.981k (0.62%)    | 13.299k (0.15%)    | \n| 100-110 | 83.066k (0.95%)    | 21.008k (0.24%)    | \n\n \n\n \n\nWe are getting the performance distribution for graphs of varying sizes (i.e., number of triples) in order to gauge whether the model generalizes well for long inputs. We will try our best to obtain and report the results within the discussion phase. Unfortunately, we couldn’t immediately fetch the results. The server room in our university that housed the GPU server used for the experiments in our submission had a recent incident which damaged the server and other servers housed there. Although we backed up the trained model, the individual sentences generated for test cases were not backed up in time. While damage assessment and recovery effort are slowly undergoing, it is unclear whether and when we could recover the data on the server. And thus we need to rerun the model on the test set on a new server. We have started doing that on a cloud computation instance.  \n\n \n\n[Colas et al., 2021] \"EventNarrative: A Large-scale Event-centric Dataset for Knowledge Graph-\tto-Text Generation.\" Thirty-fifth Conference on Neural Information Processing Systems Datasets and \tBenchmarks Track (Round 1). 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed dataset could be useful for future research for training better data-to-text generation models. The proposed method could be applied to other textual corpus and knowledge graphs. \n2. The automatic evaluation shows that pretraining on GraphNarrative and then finetuning on WebNLG effectively improved the model’s performance. The human evaluation shows that the proposed sentence trimming method effectively reduced the hallucination of the models.", "reasons_to_reject": "1. The generalization of the proposed dataset is not well studied. For example, GraphNarrative is more than 300x larger than WebNLG dataset, and it covers a lot more graph types, entities, relations and domains than WebNLG. Intuitively, models trained on GraphNarrative should be able to generalize well to WebNLG without further finetuning. However, no zero-shot results are presented. \n2. The necessity of applying the sentence trimming method is not well established. For example, in Table 3 and Table 4, the results show that training on the trimmed sentences leads to more grammatical errors in the outputs, suggesting that trimming could lead to unnatural sentences. What if you simply discard the sentences that do not align well with the graph? Given the large size of GraphNarrative, there should be a decent amount of instances left after pruning. The authors should also consider experimenting with different amounts of training data to examine the effect of training size. \n3. The comparison with recent related studies is lacking: for example, Nan et al. 21 proposed the DART dataset, which is used for studying data-to-text generation. In fact, DART could be another evaluation set for understanding the generalization of GraphNarrative. Ma et al. 22 proposed a data-to-text generation model. They also proposed to filter out instances from training data that might lead to hallucinations or missing information. The authors should consider discussing and comparing with these related works.", "questions_to_authors": "What is the average length of the sentences in your dataset? What is the sentence length distribution? If the sentences are mostly short, then the models trained on this dataset might not generalize well to tasks with long inputs.", "paper_topic_and_main_contributions": "This paper introduced a new dataset, GraphNarrative, for training graph-to-text generation models. The authors first collected Wikipedia sentences, and then find the corresponding entities in Freebase for every entity mention in the sentences. Then the knowledge triples from Freebase are further grounded to those sentences with lexical matching. The statistics show that the resulting graph-to-sentence dataset contains a much less percentage of star graphs (one central entity surrounded by other relations) compare to previous datasets. To reduce the misalignment between the graph and the sentence, the authors also proposed a method to trim sentences. They leverage the dependency graph of the sentence and try to find the shortest paths within the graph that covers all entities from the paired freebase triples. \nThe authors experimented with BART and T5 models, where they first finetune the models on GraphNarrative and then finetune on the WebNLG dataset. They also specifically compared finetuning models on original sentences vs trimmed sentences. The results show that finetuning on trimmed sentences leads to higher output quality and reduced hallucinations.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, Yangxiaokang Liu, Nadia Irwanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Mutethia Mutuma, Yasin Tarabar, Ankit Gupta, Tao Yu, et al.. 2021. DART: Open-Domain Structured Data Record to Text Generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 432–447, Online. Association for Computational Linguistics.\n\nKaixin Ma, Hao Cheng, Xiaodong Liu, Eric Nyberg, and Jianfeng Gao. 2022. Open Domain Question Answering with A Unified Knowledge Interface. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1605–1620, Dublin, Ireland. Association for Computational Linguistics."}}]

Author rebuttal:

Meta-review: This paper works on generating the natural language description for knowledge graph triples. The reviewers find the proposed datasets to be useful, practical. The proposed method seems novel and extensively evaluated with human and automatic metrics. The overall paper seems logically structured well. The reviewers share concern about not including the recent language models and not studying the decoder only models. The author responses have mostly addressed the reviewers comments and the authors should include the new results in the draft to make the paper clear. A thorough investigation of when the proposed method generalizes to good accuracy need to be worked out.
Decision: Accept-Main