FactSpotter: Evaluating the Factual Faithfulness of Graph-to-Text Generation
Authors: Kun Zhang
Oana Balalau
Ioana Manolescu
Venue: EMNLP 2023 Findings (2023)
Keywords: Graph-to-text
Factual Faithfulness
Constrained Text Generation
TL;DR: We propose a new metric for measuring the factual faitfulness of graph-to-text generations.

Abstract: Graph-to-text (G2T) generation takes a graph as input and aims to generate a fluent and faith- ful textual representation of the information in the graph. The task has many applications, such as dialogue generation and question an- swering. In this work, we investigate to what extent the G2T generation problem is solved for previously studied datasets, and how pro- posed metrics perform when comparing generated texts. To help address their limitations, we propose a new metric that correctly identifies factual faithfulness, i.e., given a triple (subject, predicate, object), it decides if the triple is present in a generated text. We show that our metric FactSpotter achieves the highest correlation with human annotations on data correct- ness, data coverage, and relevance. In addition, FactSpotter can be used as a plug-in feature to improve the factual faithfulness of existing models. Finally, we investigate if existing G2T datasets are still challenging for state-of-the-art models. Our code is available online: https://github.com/guihuzhang/FactSpotter.

FactSpotter: Evaluating the Factual Faithfulness of Graph-to-Text Generation
Kun Zhang 12 and Oana Balalau'# 1:2 and Ioana Manolescu' 1.2
Inria, IInstitut Polytechnique de Paris
Abstract
ated text, but also with its faithfulness to the input graph. While recent models such as T5 and GPT models are very fluent, they have been criticized for their factual accuracy, a problem commonly re- ferred to as hallucination (Ji et al., 2023; Liu et al., 2022). Hallucinations are serious drawback in G2T, where the generated text should only contain facts mentioned in the input graph.
Graph-to-texx (G2T) generation takes graph imput and aims to generate fluent and faith- ful textual representation of the information in the graph. The task has many applications, such as dialogue generation and question an- swering. In this work, investigate what extent the G2T generation problem is solved for previously studied datasets, and how pro- posed metrics perform when comparing gener- ated texts. To help address their limitations, we propose new metric that correctly identifies factual faithfulness, i.e., given triple (sub- ject, predicate, object), it decides if the triple is present in a generated text. We show that our metric FactSpotter achieves the highest corre- lation with human annotations on data correct- ness, data coverage, and relevance. In addition. FactSpotter can be used as plug-in feature to improve the factual faithfulness existing models. Finally, we investigate if existing G2T datasets are still challenging for state-of-tt th-the-rrt models. Our code available online: https: //gi thub. com/ httpl/uzaaannaaaSSttttttee
In this work, we focus on measuring and im- proving the factual accuracy of G2T generative models. More precisely, our contributions are as follows: i) We introduce novel metric FactSpotter for detecting if G2T generations are faithful to the facts present in the input graph; ii) We show how FactSpotter can be used in the inference step of any G2T model improve its generations; iii) We analyze the difficulty of existing G2T datasets and determine which are (resp., are no longer) challeng- ing for Ktateooftth--ae models. FactSpotter can be extended to other data-to-text tasks via methods for transforming a relational dataset into RDF, such as the R2RML language'.
1 Introduction
Related Work
Graph-to-text (G2T) generation is an important task in natural language generation, as it ren- ders graphs. and in particular knowledge graphs, accessible to non-technical users in downstream applications such as question answering (Gu al., 2021), (Romero and Razniewski, 2020), knowledge-groundee dialogue generation (Zhou et al., 2018), and document summarization (Fan et al., 2019). In recent years, there have been sev- eral datasets (Gardent et al., 2017; Nan et al., 2021) and methods proposed for G2T generation (Ke et al., 2021; Ribeiro et al., 2021), in addition to G2T competitions (Shimorina et al., 2018; Castro Fer- reira al., 2020). Evaluating text generation is challenging task in itself (Celikyilmaz al., 2020); moreover, in the context of G2T generation, we are concerned not only with the fluency of the gener-
2.1 Graph-to-text (G2T) generation
In (Ribeiro et al., 2021), the authors investigate the potential of pretrained language models (PLM) on the G2T task. They consider two Transformer- based models (Vaswani al., 2017) with an encoder-decoder structure: T5 (Raffel et al., 2020), and Bart (Lewis et al., 2020). The models receive in input a linearized (serialized) version of the input graph, in which they add the tags (H), (R), and (T") before the head entity, the relation, and tail entity of triple, respectively.
The potential of large language models is fur- ther investigated in (Keymanesh et al., 2022) on the DART dataset (Nan et al., 2021), a data-to- text dataset constructed from tables, available
Phttp://ww/r ww. org/TR//2rrl/

---

a triple-to-sentence format. The dataset is con- structed from a combination of manual and auto- matic techniques. The authors empirically evaluate the GPT2 model (Radford et al., 2019) and the T5 model on the dataset by varying the amount of su- pervision a model receives. They also investigate the potential of adding predicate descriptors in the prompt and re-ranking generations. In small-scale human evaluation, they find that their best model, T5 large, outperforms the reference text regarding hallucinations and fluency, underlining that exist- ing datasets suffer from poor human annotations, which we also observe and discuss in Section 7.
ation is the most important of these metrics. It consists of asking users to evaluate the quality
of text in a specific category, such as fluency or correctness. Unfortunately, human evaluation sometimes suffers from a low inter-annotator agree- ment (Celikyilmaz et al., 2020), (Belz and Reiter, 2006) as different people might have different no- tions of what makes a text fluent or correct, or the instructions they receive for annotation might lack sufficient clarity. Also, human annotation is time and money-consuming: can represent a bottleneck in the iterative process of improving a model. Hence the need for automatic metrics such as BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), METEOR (Banerjee and Lavie, 2005), BERTScore (Zhang et al., 2019), BLEURT (Sellam et al., 2020) BARTScore (Yuan et al., 2021), In- foLM (Colombo al., 2022), among others. Over- lap measures based on n-grams, such as BLEU, ROUGE, and METEOR, have been widely used in the literature, while recently proposed metrics based on word embeddings, such as BERTScore, BLEURT, BARTScore are gaining traction. The embedding- umbeeddigg-based measures have been shown to correlate better with human evaluation than the n- grams metrics. In addition, some metrics, such BLEURT, have been trained on human annotations. As these automatic metrics are based on distances or similarities to ground-trute trunh texts, they rely on the quality of annotated sentences.
The authors of (Ke et al., 2021) propose modify- ing the Transformer model by adding extra atten- tion and pooling layers to improve G2T generation. In addition, the model's pretraining has three steps given the input graph and the expected output text: 1) reconstructing the text sequence given the com- plete subgraph, 2) predicting the masked entities and relations in the corrupted subgraph, given the complete text, and, 3) aligning the embedding vec- tors of the knowledge graph and the text.
Most of-te-of-thh-art G2T models are based on Transformers, and they can generally generate flu- ent texts related to given graphs. Although various baselines designed neural networks encode both global and local information (Ribeiro al., 2020), they cannot guarantee that generated texts are fac- tual faithful to the given graphs. It's also not clear whether current G2T datasets are still challenging.
2.2 Evaluation metrics for text generation
Apart from the works mentioned above, a few prior studies assessed factual faithfulness in graph- to-text generation. In (Faille et al., 2021) the au- thors introduce metric for verifying whether en- tities in input graphs are represented in the gener- ated texts. However, this work does not evaluate the quality of the predicates the generated texts, which is much more difficult task. In (Rebuffel et al., 2021), a question generation (QG) and ques- tion answering (QA) framework is employed to evaluate the quality of the generated text by deter- mining if each question posed by the QG module can be addressed by the QA module. We believe our contribution can further advance the sfate-of- the-art as: i) FactSpotter requires significantly less computational resources; ii) FactSpotter is self su- pervised, thus it does not requires additional data to the G2T model; item FactSpotter can be pluged into a G2T to improve its generation.
Alongside the significant improvements that mod- els for G2T generation underwent and, in general, the improvement of language models, new metrics to assess the generations" quality have been pro- posed (Celikyilmaz et al., 2020; Sai et al., 2022). G2T generation belongs to the broader field of natu- ral language generation, including tasks such as ma- chine translation, automatic summarization, ques- tion answering, and more. Each task has specific re- quirements, which might entail using some metrics over others. For example, in machine translation, the translation should match the ground-truth text as closely as possible, while in chatting or summa- rization, adding or removing some information is acceptable or even desirable.
Evaluation metrics are split into three categories: human-centinn metrics, untrained automatic met- rics, and machine-leamel metrics. Human evalu-

---

3 Problem Statement
(subject, predicate, object) and a text T, it assigns a score between and 1, where score close to 0 sig- nifies that the text that does not correctly represent the fact, and a score close to rewards a factual faithful textual representation of f in T. Such a metric can be used to compare different generation systems and. in addition, assess single system on given dataset, to determine how close the system is to representing the input graphs correctly.
A knowledge graph (KG) consists of facts (or data triples) of the form (subject, relation, object/literal) and/or an ontology deseribing the properties that hold between classes and/or relations.
Graph-to-text. G2T tool takes in input graph and outputs textual representation of the graph. G2T inputs are often subgraphs of larger real-world graphs, such as knowledge graphs. The textual rep- resentation should be fluent and should contain all the facts present in the input graph. For example, given a subgraph consisting of the single DBpe- dia (Auer et al., 2007) fact (The Myth of Sisyphus, author, Albert Camus), we would like to gener- ate the sentence The Myth of Sisyphus was written by Albert Camus. This work primarily focuses on creating textual representations of KG subgraphs.
The intuition of our score is the following. We train a model to perform a task simpler than G2T generation: only detecting whether facts exist in a sentence and whether they are well expressed. This simpler model can then be used as a plug-in fea- ture by any existing G2T model. aid perform the more complex task of language generation.
Our metric, FactSpotter, is trained as binary classifier. Given in input fact (subject, predicate, object) and sentence, it should predict 1 if the fact is well expressed in the sentence, 0 otherwise. Thus, FactSpotter is inherently interpretable. We leverage as classification models recent large lan- guage models, capable of detecting semantic close- ness, even different words, c.g., synonyms, are used. This approach is similar to the one taken to compute metrics such as BertScore and BartScore. Given an input G2T dataset D, with a training (train) set, a development (dev) set, and test set, create the training set as follows:
Factual faithfulness. The following human cri- teria have been proposed to evaluate the factual quality of a generated text (Castro Ferreira et al., 2020): data correctness, data coverage, and rele- vance. Given graph G' and machine-generatee text T, the generated text is characterized by:
1. Data coverage or recall: are all the descrip- tions presented in the graph G included in the text T? i) Covered predicates: does T contain all the predicates from G? ii) Covered entities: does T contain all the entities from G?
Positive samples. Given an instance of the train- ing set of the form (graph G, ground truth text T), for each fact (triple) in G, we create a positive sample of the form (fact f. ground truth text T)).
2. Relevance or precision: i) Relevant predi- cates: does T only contain relevant predi- cates? ii) Relevant entities: does T only con- tain relevant entities?
Negative samples. Given an instance of the train- ing set of the form (graph G, ground truth text T), for each fact f E G, we create negative sam- ples as follows: i) Type I: we perturb the fact f: we change its predicate, or an entity (subject object), or both, while the ground truth text T remains un- changed. ti) Type II: perturb the ground truth text T: we drop one or both entities related f from the text, or drop the n-grams most similar to the predicate of f. or we apply simultancously several modifications, keeping the fact unchanged. For example, given the fact (The Myth of Sisy- phus, author, Albert Camus) and its associated text "The Myth of Sisyphus was written by Albert Ca- mus", a Type I negative sample alters the fact ((The Myth of Sisyphus, author, Simone de Beauvoir). "The Myth of Sisyphus was written by Albert Ca-
3. Correctness: are predicates correctly men- tioned and adequately introduced in the data?
Research questions. We focus on the following three research questions:
RQ1 What metric would better correlate with the factual faithfulness of generated text?
RQ2 Can we improve factual faithfulness of G2T?
RQ3 Is the G2T task solved on existing datasets?
4 FactSpotter: An explainable metric for factual faithfulness
In this section, we introduce a new metric for fac- tual faithfulness. A good metric should be inter- pretable, that is: given in input a fact of of the form

---

mus"), while for Type II yields the sample (The Myth of Sisyphus, author, Albert Camus), "The Myth of Sisyphus was written"). We associate probabilities to each perturbation, and control the generation such that for each positive sample, we only generate one negative sample. To allow our classifier to learn from different negative samples and avoid over fitting (Chen et al., 2020), for each training epoch, we use newly generated set of negative samples. The development set is built in the same way. Through evaluation on fixed test set (Appendix 1.4.1) we find that the model which best detects factual faithfulness is the one with the highest probability of perturbing the fact (90%) and 10% the probability of perturbing the ground truth.
tual faithfulness of the generated text? For this question, we compare FactSpotter with: BLEU (Pa- pineni et al., 2002), METEOR (Banerjee and Lavie, 2005), BERTScore (Zhang et al., 2019), BLEURT (Sellam et al., 2020), BARTScore (Yuan et al., 2021). The only metric that is not normalized is BARTScore. This metric has variant specifi- cally adapted for factual faithfulness, BARTScore- faithful. Further details of these metries are pro- vided in Appendix A.I.
We calculate the system-level correlation be- tween automatic metrics and human annotations. Given S systems under evaluation, for a certain dimension, e.g., fluency, compute the correla- tion between the system-level automatic metric scores (M(S)).... (Ss)) and the correspond- ing system-level human scores (S)),, (Ss)), where (S.) is the score of the automatic metric on the texts from system S,, and H (S.) is the score of the human annotation on the same result. Simi- larly to (Colombo et al., 2022), we compute three correlation metrics: Pearson correlation (r), Spear- man correlation (p), and Kendall's Tau (r). To test if metric M1 has a higher correlation with human annotations than M2, we use the bootstrap- ping technique proposed in (Wilcox, 2016), which we describe in the Appendix A.2. We also report the sentence-level correlation in Appendix A.5. In addition to automatic measures, we report the corre- lation between one annotator and the average score of the remaining annotators, which should be up- per bound on the correlation we can obtain using automatic measures.
Above, we have described tactSpotter (trained) classifier. To use it as a score (metric), we take the output of the model after the softmax layer. The final score of a generated text T given an input graph G is the average over the scores for each pair (fact/ ʃ E G. generated text T).
Parameters and performance of FactSpotter. As we aim to add our metric, FactSpotter, in the inference step of graph-to-text generation, we pre- fer small language models. Hence, we select the small Electra model (Clark et al., 2020). We have experimented with other small models, such as DistiBBERT and BEER,, but we did not observe an improvement. We train our classifier for 16 epochs, with a learning rate of 10 and the Adam optimizer. We describe in the Appendix A.4.1 how we chose the percentages of negative samples for FactSpotter.
The performance of FactSpotter on the test splits across multiple datasets is detailed in Table 1, with accuracy and Fl score. We also report numbers of true positives/negaativs (TP/TN), and false posi- tives/negatives (FP/FN) in the table.
WebNLG 2017. In the WebNLG 2017 challenge (Shimorina al., 2018), the organizers annotated submissions on semantic adequacy (the text cor- rectly represents the meaning in the data), text struc- ture (as above, referred in the original paper as grammar) and fluency (as above). This annotation has carried over 223 samples.
Dataset Acc.
F1
TP/TN
FP/EN
GrailQA
96.85
96.55
2272/2643
64/96
SimpleQ.
95.41
95.41
10260/10438
419/575
DART
97.62
97.62
26613/26352
607/684
WebNLG 2020. After the WebNLG 2020 Chal- lenge (Castro Ferreira et al., 2020), the organizers annotated the 16 participating systems on data cor- rectness (the predicates found in the data are cor- rectly mentioned together with their subject and ob- ject), data coverage (the text includes descriptions of all predicates presented in the data), and rele- vance (the text describes only those predicates with related subjects and objects which are in the data), in addition to text structure (the text is grammati-
WebNLG17
99.10
99.09
8594/8575
93/63
WebNLG20
95.21
95.21
9637/9853
317/663
Table 1: Performance of FactSpotter on test splits.
5 Evaluating graph-to-text generation
We investigate our first research question (RQ1): which metrics would better correlate with the fac-

---

Correct.
In this section, we investigate the answer to our D. Cover.
Relev.
Fluency
T. Struet. p
Metric
p
7
p
p
Correct.
1.0
1.0
1.0
0.96
0.81
0.66
0.97
0.81
0.66
0.80
0.77
0.60
0.79
0.76
0.59
D. Cover.
1.0
1.0
1.0
0.93
0.80
0.64
0.71
0.56
0.43
0.69
0.55
0.42
Relev.
1.0
1.0
1.0
0.76
0.63
0.48
0.76
0.62
0.47
Fluency
1.0
1.0
1.0
0.98
0.97
0.91
T. Struct.
1.0
1.0
1.0
Human
0.96
0.80
0.65
0,93
0.83
0.68
0,96
0.74
0.59
0.95
0.93
0.80
0.93
0.91
0.77
BLEU
0.59
0.64
0,48
0.53
0.53
0,40
0.56
0.60
0.45
0.87
0.84
0.68
0.86
0.84
0.68
METEOR
0.72
0.75
0.60
0.65
058
0,44
0.70
0.64
0,50
0.88
0.89
0.74
0.86
0.88
0.72
BERIFI
0.83
0.77
0.60
0.74
0.58
0.43
0.81
0.65
0,50
0.90
0.93
0.80
0.88
0.92
0.78
BLEURT
0.93
0.82
0.67
0.86
0.65
0.50
0.91
0.69
0.55
0.90
0,92
0.78
0.89
0.91
0.76
BARTS
0.90
0.83
0.67
0.86
0.71
0.53
0.88
0.71
0.56
0.77
0.81
0.63
0.75
0.80
0.62
BARTS-F
0.67
0.54
0.41
0.68
0.61
0.46
0.68
0.59
0.45
0.51
0.52
FactS
0.94
0.80
0.64
0.91
0.87
0.71
0.96
0.79
0.64
0.74
0.59
0.45
0.72
0.59
0.45
Table 2: Correlation at the system level with human judgment correctness, data coverage, relevance, fluency and text structure for the 2020 WebNLG task. For tables here and below, BERTFI stands for BERTScore-Fi FactS for FactSpot ter. We highlight the best result
BARTS for BARTScore, BARTS-F for BARTSScore- and we mark it with asterisk when it is statistically significantly larger than any other metric (excluding Human to Human correlation). Fact ter performs the best on correctness, data coverage, and relevance. We put a value for correlation if the pvalue p 0.05.
Adoq.
Struet.
Fluency
Improving the factual faithfulness of graph-to-text generation
Metric
e
Som. Addy
100
T.
1.0
0.73
0.63
0.32
0.71
0.66
0.32
Struct.
1.0
1.0
1.0
0.98
0.95
0.88
Flunncy
1.0
Human
0.99
0.95
0.94
035
0.92
0.84
0.95
0.79
BLEU
0.76
0.71
0.36
0.56
0,71
0.57
0.83
0.72
0.57
METEOR
0.86
0.83
0.67
0.85
0.70
0.57
0.80
0.70
0.56
third research question: can we improve graph-to-
DERIT
0.70
0.57
BLBURT
0.90
0.88
0.72
0.84
0.69
0.56
0.79
0.68
0.56
text generation on factual faithfulness (RQ2)? For this, we first explain how to improve the inference step of any G2T' model using FactSpotter, and then present the results of this technique on the state- of-the-art models for G2T generation.
BARTS
0.90
0.87
0.78
0.71
0.68
Facts
0.97
0.93
0.33
0.67
Table 3: Correlation at the system level with human
judgment on semantic adequacy, grammar, and fluency. for the 2017 WebNLG dataset.
6.1 Improving models' factual faithfulness
cal, well structured, and written in good English) and fluency (the text progresses naturally, forms a coherent whole and is easy to understand). 178 generations of each system were annotated.
Let M be neural network G2T (seq2seq) model that, given an input sequence æ II, I1, TM. produces an output sequence y y1, 12.32,.... yN. where 24 E Vg. V/i E Vy, and Vg. Vy are the vo- cabularies from where we select items in the input, respectively output sequence. In the inference step. the model generates the sequence y that maximizes:
Results. On WebNLG 2020, Table 2 shows
that FactSpotter has the best performance on fac- tual faithfulness, significantly improving relevance. BLEU, METEOR, BERT and BLEURT reach similar fluency and text stucture scores. For the results on WebNLG 2017 in Table 3, FactSpotter has the highest performance on semantic adequacy, which is the only dimension related to factual faith- fulness. For text structure and fluency, BLEURT obtains the best results, although the results not statistically significant. Overall, previous met- rics are better on text structure and fluency, which are generally considered as no longer a bottleneck for large language models. FactSpotter is the best suited metric on factual faithfulness.
N
P(ula) = II P(ullues, E=
In practice, for computational efficiency, the log of the probabilities are typically utilized in beam search. We use the following method to improve factual faithfulness in G2T inference without re- training the model.
Given:i) graph-to-text generation model M, ii) our factual faithfulness classifier, i.e., FactSpotter, iii) a subgraph G composed of facts, we encourage factual generations by modifying the

---

prediction step as follows:
the weights with a pretrained T5 base model of- fered by the authors of JointGT. For the fine-tuning step (each model is fine-tuned on the training split of the datasets), we train the small models with a learning rate of 10 4 and a batch size of 32, while the base models are trained with a learning rate of 5 10-5 and batch of 16. We use beam size of 5 and the AdamW optimizer for both sizes of mod- els. For Equation 1, we fix the weight À 0.15 (parameter tuning in Appendix A.4.2).
log( P' (yilU<ss
11 Prace, (yes 1)) log Paac, (yes)
+ log( ((u) Ueci, x))
(1)
where: i) pt ()) is the probability of generating token y/ given the factual classifier; ii) Pract (U<i 1) is the probability of correctly representing the fact j in the previously gener- ated tokens yo: y/- computed by FactSpotter. ii P( is the probability for generating the next token based on previous i - tokens.
6.3 Datasets
To evaluate G2T performance, need (graph, text) pairs datasets. The graphs can be directly extracted from an underlying knowledge graph or adapted to have richer semantics, such as query graphs (Yih et al., 2015). The text associated to the graph should be a sentence or a paragraph verbaliz- ing all the information contained in the subgraph. Several datasets are proposed in the literature for the G2T task, such as WebNLG (Gardent et al., 2017). Many question-answeering (QA, in short) datasets are also in the form (graph, text). Since question answering datasets can be very large and cover many KG predicates (Gu et al., 2021), we also leverage such datasets. To ensure that FactSpotter has never encountered the test data, it is trained exclusively on the training set and eval- uated it on the validation split.
In our equation, a fact j is encouraged only if we have not observed in the text generated until step i, according to Pact, (V<i 1). Then adding token I would increase the probability of the text includ- ing the fact j. Prect, (y<s). When Pfact, (yes 1) is small, the equation encourages the selection of words belonging to the fact j. As is large and 1 - -Paaa, (1<1-) tends to 0. then we can con- sider that the fact j already appears in the text, and words that satisfy fact j will no longer be en- couraged. The weight À controls the influence of the FactSpotter on the prediction of the following words. A high À might yield factual text, but not necessarily a fluent one. We generate tokens till we have generated text S yoo yk for which Vj E F, Pact,, (S) > 0.5, i.e., the probability of each fact j in G is verbalized in the text is over 0.5, the standard positive threshold.
SimpleQuestions (Bordes et al., 2015) is QA dataset built on Freebase (Bollacker al., 2008). The dataset contains 108K (triple, question) pairs, where the question corresponds to the subject and predicate from the triple, and the answer is the ob- ject of the triple. For example, given the triple (Gul- liver's Travels, book/written work/author book/written_worrkuutho: Dean Swift), the question is Who wrote Gulliver's Trav- els?, with the answer Dean Swift. The dataset covers 76 domains, 741 classes, 89K entities and 2K relations. A Freebase domain a general area of knowledge such as business, politics, economics, etc. We created our own split for this dataset, where the test set is zero shot: we have not seen the pred- icates during training. FactSpotter can be trained to correctly classify if question refers to a triple, even if the object or subject is missing from the question, as we replace the entity with its type.
6.2 Models
We consider for our evaluation state-of-the-ar mod- els for G2T genetation, PLM (Ribeiro et al., 2021) and JointGT (Ke et al., 2021). The former investi- gates how standard seq2seq model can perform on G2T, given a carefully constructed representa- tion of a graph. The latter proposes more com- plex neural network, with built-in attention layers specialized on graph structured inputs. Both are ini- tialized with the pretrained weights of a language model. Similar to the authors, we consider in this work T5 (Raffel et al., 2020) for both G2T models. For simplicity and to the second model as JointGT. We refer to the models modified as explained in Section 6.1 as FactT5 and FactJointGTT. For T5 and FactT5, we initialize the weights with T5 small, and T5 base. For Jo and FactJointGT we initialize
refer to the first model T5,
GrailQA (Gu et al., 2021) is also a QA dataset that uses Freebase, created using human annota- tion. The original dataset contains 64K (triple, question) pairs, however, the test set is not released

---

the authors have created a QA challenge hence we use the development set as set. The re- maining data (training and development) consists of 51K pairs. The authors propose three levels of generalization and split the development and test as follows. 50% of the pairs from held-out domains (Freebase assigns to each entity and predicate a do- main, such as music, sports, etc.) are not covered in training: this is the zero-shot split. 25% of the pairs correspond to graphs where the combination of ontology items (classes and predicates) were not covered in training: this is the compositional split. Finally, the remaining 25% are randomly sampled from the same distribution as the training dataset: the i.i.d. split. The i.i.d. and compositional subsets contain only ontology items (classes and predicates) covered in training. For the zero-shot subset, five domains are held out for validation.
thors align the predicates such that predicates with the same meaning have the same representation. The dataset has 82K instances. We excluded the WikiSQL split as it has been generated automati- cally and after we performed a manual verification, we observed many low quality ground truth texts.
6.4 Evaluation
In this section, we consider (RQ2): Can we im- prove G2T generation on factual faithfulness?
Table 4 shows the results the SimpleQues- tions dataset. We generally have a high FactSpotter score, indicating that models are already good relaying factual information. We can improve the factual faithfulness with F-T5 and FGT without significant compromise on other metrics, implying maintained fluency of texts.
BLBU
METEOR
BLEURE
BARTS
FactS
TSS
37.97
36.50
93.43
67.85
-2.45
96.80
WebNLG (Gardent et al., 2017; Castro Ferreira al., 2020) is text generation dataset on DBPedia, created via human annotation. The dataset consists of (graph, paragraph) pairs, where the graph is a set of DBPedia facts, and the paragraph consists of one or several sentences that describe the graph. We use the 2017 (2.1) and 2020 (3.0) versions of the dataset? 3. The 2017 version of the dataset con- tains 42K graph-text pairs, and has two splits, the standard version and the constrained version. In the constrained version, the test set does not contain that a triple occurring in train/dev. In this work we considered only the constrained split, as it is more challenging. The WebNLG 2020 dataset has 40K pairs, which comprises 10 categories that were previously seen and utilized in WebNLG 2017, as well as 5 categories that were unseen in WebNLG 2017 are now incorporated into the seen data of the WebNLG 2020 dataset. It also introduces a new category of data: company.
F-T5S
37.85
36.06
93.42
67.86
-2.45
98.17
TSB
38.73
36.43
93.61
68.48
-2.43
95.09
38.73
36.42
68.42
-2.43
97.14
JGI-TS
39.35
36.82
93.65
68.50
-2.42
95.40
FGPT5
39.24
36.78
93.64
68.48
-2.42
97.25
Table 4: Results on G2T on SimpleQuestions. Here and below, T5S stands for T5 small, TSB for T5 base, F-T5S for FactT5 small, F-T5B for FactT5 base, JGT-T5 for JoinIGT-T5, and FGT-T5 for FactJointGT-TS.
Model
BLBU
METEOR
BERIPI
BLEURE
BARIS
TSS
66.24
47,80
96.72
73.16
-1.41
98.67
F-138
66.27
47.89
96.73
73.21
-1.41
99.25
TSB
67.04
48.35
96.81
73.22
-1.40
99.44
F-TSB
67.04
48.22
96.80
73.26
-1.40
99.71
JGT-TS
67.08
48.34
96.76
73.44
-1.39
99.09
FGT-T5
66.89
48.19
96.84
73.42
-1.39
99.67
Table 5: Results on G2T on WebNLG 2017 Const.
METEOR
BLEURE
BARTS
Facts
TSS
52.30
40.82
95.43
-1.75
65.80
90.75
F-138
52,44
41.02
93.45
-1.74
63.92
93:45
TSB
54.29
41.66
93.65
-1.69
66.43
93.60
54.72
41.70
-1.69
66.46
DART (Nan et al., 2021) is a data-to-text dataset based on Wikipedia tables. Since the tables are represented as (subject, predicate, object) triples, it also suits our evaluation. Besides creating table- to-text annotations, the authors also existing datasets: the QA dataset WikiSQL. (Zhong al., 2017), the cleaned E2E (Dušck al., 2019) (entity- to-entity relations the restaurant domain), and the original release of the WebNLG dataset for the 2017 challenge (Shimorina et al., 2018). The au-
JGT
54.23
41.49
93.47
-1.72
66.23
91.26
FGT-S
$4.45
41.52
93.49
-1.72
66.31
93.16
Table 6: Results on G2T on the WebNLG 2020 dataset.
BLRU
MRTFOR
BERTFI
BLFUUT
BARTS
Facts
TSS
46.22
39.96
94.69
56.62
-2.03
95.47
F-TSS
46.31
40.07
94,74
66.66
-2.02
97.29
TSB
48.47
40.74
95.04
67.49
-1.97
96.65
40.72
67.43
-1.97
97.60
JGT TS
47.51
40.43
94,92
67.33
22011
95.86
FGT
47.39
40.32
94.92
67.26
-2.00
97.25
Table 7: Results G2T the DART dataset.
Phttps: //dki-lab. watekiiilabbgthhheee wwtttb.ii//raiilll.. io/Grail0A/ Phttps: htt.//tttaata//////////ee/e..a httpg//aaacoo/o/ooo.

---

Medel/Split
BLEU
MCTEOR
BERTEI
BLEURT
BARTS
Facts
tional split, we have 13 improved generations, but also 5 that are less factual. For the IID split, 4 generations are improved, others are rephrased texts. Only 7 samples of IID improve over 0.01 for FactSpotter, so this split is not challenging. For on DART has false positives. For WebNL. 2017 dataset, 11 generations are more factual, others are rephrased texts. WehNL.G 2017 is only has 16 generations improve over 0.1 for FactSpotter,
ID
TSS
44.51
-2.37
T-T5S
44.64
41.88
93.25
69.63
2.36
98.47
TS8
45.95
42.71
93.50
70.66
-2.29
99.43
F-T5B
46.10
4266
93.52
70.73
-2.29
99.50
JGT-T5
43.68
41.65
93.21
69.41
-2.37
98.62
PGTTS
43.61
93.19
-2.37
99.12
the DART dataset, texts are more factual. DART dataset has samples that ground-truth sentences do not match with graphs, so FactSpotter trained
Zero
TSS
30.30
36.91
91.74
62.87
-2.74
93.27
F-T55
30.30
36.90
91.75
63.01
-2.73
94.60
TSD
32:20
37.35
91.92
63.84
2.772
94.77
F-T5B
32.39
-2.71
95.61
JGT:T5
32,94
37.69
92.02
64.18
-2.68
94,15
FGT-TS
32:46
37.55
91.93
64.00
-2.68
94.95
Comp.
TSS
30.38
35.32
92.09
63.99
.2.74
94,94
F-T58
-2.75
whose baseline is very high. For WebNLG 2020,
T5D
31.75
35.64
92.24
63.99
.2.72
94,84
F-T5B
31.66
92.24
-2.71
12 generations are more factual, and 3 are rephrased 95.26
96.53
JGT-T5
31.46
36.08
92.39
64,92
2.67
texts. 5 generations in WehNLG 2020 have higher FactSpotter than baseline generations, but they'r re still not factual enough.
TGT
31.25
36.21
92.43
63.12
-2.65
97.10
Table 8: Results on G2T on the GrailQA dataset.
For the cases where a generation becomes less factual, this is a consequence of the accuracy of FactSpotter, which we present in Section 4. Given that our metric does not correlate strongly with fluency, we perform second analysis on genera- tions observe if there is decrease in fluency. To answer this question, we study the top 20 sen- tences for which the BLEURT decreased the most in comparison with the original generated question. We do not observe an obvious decrease in fluency on any dataset, the decrease in BLEURT score is due to several other factors: BLEURT has diffi- culties identifying rephrased sentences, in few cases the factual faithfulness decreased, and in the remaining cases the generations are more faithful to the input graph than the ground truth sentences. however BLEURT cannot identify it. Hence, we can conclude that adding FactSpotter as a plugin in generation can improve G2T generations on factual faithfulness and does not affect the fluency.
Table 5 has the highest FactSpotter score from all datasets, which means that we observe the most factual generations on WebNLG 2017, with F-T5 and FGT having slightly higher scores.
In Table 6, the FactSpotter scores are lower for the WebNLG 2020 test split, although achieve scores comparable to WebNLG 2017 on its vali- dation split. This discrepancy may be attributed to the difference in distribution between the test and training splits of WebNLG 2020. F-T5B can achieve higher FactSpotter score than T5B without compromising fluency. We observe the same trends for the DART dataset in Table 7.
In Table 8, all the metrics are higher for the IID split of GrailQA, and in particular FactSpotter can reach 99.5%. hence the models learn to reproduce triples seen in training. For shro and Com- positional splits, larger models are better, and our factual inference improves the score of FactSpotter. We illustrate some improved samples of Zero-shot and Compositional splits in Appendix A.3, and in Appendix A.6, we investigate the impact of varying numbers of triples in the input subgraphs on the quality of the generated text.
7 Remaining challenges in G2T task
Finally, we consider (RQ3): is G2T task solved on existing datasets? We have observed high FactSpotter scores in Section 6 on the performances of models. We use FactSpotter to do a more de- tailed analysis: we investigate what is the percent- age of generations in each dataset which had at least fact considered missing by FactSpotter. A fact is considered missing if the score of the pair (fact, generated sentence) is lower than 0.5. We obtain the following statisties: 1.94% of texts miss a fact in SimpleQuestions; 7.27% of texts miss at least a fact in DART; 5.79% of WebNLG 2017 texts
To validate that indeed generations improve us- ing FactSpotter in inference, we select the best FGT-T5 model and we analyse the top 20 phrases where the FactSpotter score improved the most compared to the JGT-T5 generations.
For the SimpleQuestion datasets, we have 15 generations that are more factual, and 5 generations less factual. For the Zero-shot split of GrailQA, 14 generations are more factual. For its Composi-

---

miss least one fact, and 12.64% for WebNLG 2020; For GrailQA we have 5.8% for the zero shot split, 1.36% for the compositional split and 1.13 for the IID split. According to the observations, WebNLG 2020 is the most challenging dataset, fol- lowed by DART, the zero shot split of GrailQA, and the WebNLG 2017 dataset. In Appendix A.7. taking GrailQA and WebNLG 2017 as examples, we analysed the difficulty of G2T datasets from different knowledge graphs, by looking into how often predicates and entity names are rephrased or expressed exactly as in the input graph.
is 73.44%. The BLEURT scores of the generations vary from 0.46 0.99; more than 50% of the test set generations score less than 0.80. We sampled 40 generations with BLEURT score < 0.8 and note that 35 generations are correct, which are rephras- ing the ground truth, while 2 out of 35 that are bet- ter than the ground truth. Hence, we observe that BLEURT score cannot be used to determine if we have achieved a good performance can only be used to compare different models. This issue has also been pointed out by the authors".
dataset, it
FactSpotter answers whether a fact is present text; does not have to address the much harder task of deciding if two sentences are equivalent. Besides being more reliable because it is solving a simpler task, it is also more interpretable as we can investigate the exact triples that classified negative, instead of doing complete comparison between subgraph and a sentences or between two sentences. This is especially useful for long input graphs and long generations.
We perform a second evaluation, this time by manually analyzing the output of the models. We consider the worst 20 sentences according to BLEURT and FactSpotter, hence 40 examples per dataset or split. On SimpleQuestions, the genera- tions are fluent, however 22/40 have an incorrect predicate. For GrailQA, in the IID split the predi- cates are correctly generated, but the models still have difficulties in generating some entity names (16/40). For the sero split, generations suffer from wrong entities and predicates (22/40). The compositional split has several samples with wrong ground truth (6/40 of the worst generations) and 19 out of 40 incorrect generations. For DART, 24 generations are not correct. On WebNLG 2017, from the worst 40 generations, only two might benefit from improved fluency, while in many ex- amples, the generated sentence was more fluent than the ground truth (14/40). Regarding correct- ness, only out of 40 generations had a missing triple, while two generations incorrectly used a predicate. On WebNLG2020, only one instance exhibits room for improvement in fluency, but 24 instances either omit factual information con- tain incorrect facts. Among the 20 outputs with the lowest BLEURT scores, are rephrased texts with correctly explained facts. In contrast, among the 20 outputs with the lowest FactSpotter, only instances fall into this category.
Conclusion
In this work, we have presented new metric for measuring factual faithfulness in G2T, FactSpotter, This metric can be trained self supervised fash- ion, using the same annotations as a G2T model. We have shown it achieves the highest correlation with humans on factual faithfulness and it can be used as a plug-in feature for G2T models. Finally, we have used our metric to analyze the difficulty of existing datasets. We have observed that models perform very well these datasets, hence new datasets should be proposed. Such datasets could cover more difficult input graphs, for example triple from tables. addition, through the initiative governments to release tabular data related to pub- lic interest tools trained to express in natural lan- guage the content of tables could be used as user friendly interfaces for citizens.
Acknowledgment. This work was performed us- ing HPC resources from GENCI-IDRIS (Grant 2023-AD01 1014244). The authors were partially funded by the ANR-20-CHIA-00015 project.
Based on our manual annotations, we observed that models are able to produce correct genera- tions. However, when the generation is rephrased in respect to the ground truth sentence, metrics like BLEURT, which measure if two sentences equivalent, struggle assign high scores. We recall that BLEURT, a normalized metric, gives score of equivalent sentences. On the dataset WebNL.G 2017, our metric assigns a very high score to the models, while the highest average BLEURT score
issues/1
FFor example, Eurostat, https::/ec.europa..eu

---

9 Limitations
Anja Belz and Ehud Reiter. 2006. Comparing auto- matic and human evaluation of NLG systems. In 11th Conference of the European Chapter of the As- sociation for Computational Linguistics, pages 313- 320, Trento, Italy. Association for Computational Linguisties.
Our work has the following limitations:
FactSpotter cannot be used to determine the precise nature of the error in the generated sentence. It was trained to predict whether a fact is presented in the text or not, not if we the sentence has wrong predicate or wrong subject object. This problem can be solved by a second classification step for whether predicates or entities are incorrectly verbal- ized in the text, to make FactSpotter more in- terpretable.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: A col- laboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, SIGMOD '08, page 1247-1250, New York, NY, USA. Association for Computing Machinery.
Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. 2015. Large-scale simple question answering with memory networks. arXiv preprint arXiv:150.002075
The input of FactSpotter is the concatenation of a fact represented in triple and natural language text T, i.e., has limited input for- mat. With such input, the advantage is that it is casier to construct both positive and negative samples for its self-supervised training. How- ever, it is also difficult to use it to check the factual faithfulness on other text generation tasks, since high quality structured knowledge graphs are hard to generate. However, we will investigate in the future the use of open in- formation extraction models (Upadhyay et al., 2023) for extracting facts from sentences.
Thiago Castro Ferreira, Claire Gardent, Nikolai Ilinykh, Chris van der Lee, Simon Mille, Diego Moussallem, and Anastasia Shimorina. 2020. The 2020 bilingual, bi-directional WebNLG+ shared task: Overview and evaluation results (WebNL.++ 2020). In Proceed- ings of the 3rd International Workshop on Natu- ral Language Generation from the Semantic Web (WebNLG+) ), pages 55-76, Dublin, Ireland (Virtual). Association for Computational Linguistics.
Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: A survey, arXiv preprint arXhv:2006. 14799.
Although the accuracy and Fl score of our classification model on the test splits of var- ious datasets in Table are high, there still exist some false positive and false negative samples. Hence, our FactSpotter generally re- flects factual faithfulness, but it might still be biased on some hard samples, especially when predicates in knowledge graphs are distant to their natural language representations in the vector space of language models.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In In- ternational conference on machine learning, pages 1597-1607, PMLR.
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020, Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv: 10555.
Pierre Jean A Colombo, Chloé Clavel, and Pablo Pi- antanida. 2022. InfoLM: A new metric to evaluate summarization & data2text generation. In Proceed- ings the AAAI Conference on Artificial Intelligence, volume 36, pages 10554-10562.
References
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. Dbpedia: A nucleus for web of open data. In The Semantic Web: 6th International Semantic Web Conference, 2nd Asian Semantic Web Conference, ISWC 2007+ ASWC 2007, Busan, Korea, November 11-15, 2007. Proceedings, pages 722-735. Springer.
Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming- Wei Chang, Dipanjan Das, and William Cohen. 2019. Handling divergent reference texts when evaluating table-to to-ttotteeet generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4895, Florence, Italy. Asso- ciation for Computational Linguistics.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with im- proved correlation with human judgments. In Pro- intrinsic and ex-
Ondrej Dušck, David M. Howcroft, and Verena Rieser. 2019. Semantic noise matters for neural natural lan- guage generation. In Proceedings of the 12th Interna- tional Conference Natural Language Generation, pages 421-426, Tokyo, Japan. Association for Com- putational Linguistics.
ceedings of the acl workshop trinsic evaluation measures for machine translation and/or summarization, pages 65-72.

---

James D Evans. 1996. Straightforward statistics for the behavioral sciences. Thomson Brooks/Cole Publish- ing Co.
BART: Denoising suueenceto-sequenee pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguisties.
Juliette Faille, Albert Gatt, and Claire Gardent. 2021. Entity-based semantic adequacy for data-to-text gen- eration. In Findings of the Association for Computa- lional Linguistics: EMNLP 2021. pages 1530-1540, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Chin- Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text summariza- tion branches out, pages 74-81.
Angela Fan, Claire Gardent, Chloé Braud, and Antoine Bordes. 2019. Using local knowledge graph con- struction to scale Seq2Seq models to multi-document imputs. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLA) pages 4186 4196, Hong Kong. China. Association for Com- putational Linguistics.
Tianyu Liu, Yizhe Zhang, Chris Brockett, Yi Mao, Zhifang Sui, Weizhu Chen, and Bill Dolan. 2022. A token-level reference- hallucination detection benchmark for free-form text generation. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 6723-6737, Dublin, Ireland. Association for Computational Linguistics.
Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsich, Xi- angru Tang, Aadit Vyas. Neha Verma, Pranav Kr- ishna, Yangxiaokang Liu, Nadia Trwanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Mutethia Mutuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Cher Tan, Xi Victoria Lin, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. 2021. DART: Open- domain structured data record to text generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 432-447, Online. Association for Computa- tional Linguistics.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Brezz-Beltrachini. 2017. Creating training corpora for NLG micro-planners. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 179-188, Vancouver, Canada. Association for Computational Linguistics.
Yu Gu, Sue Kase, Michelle Vanni, Brian Sadler. Percy Liang. Xifeng Yan, and Yu Su. 2021. Beyond i.i.d.: Three levels of generalization for question answer- ing knowledge bases. In Proceedings of the Web Conference 2021, WWW 22,, page 3477-3488, New York, NY, USA. Association for Computing Machin- ery.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. BLEU: a method for automatic eval- uation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- rional Linguistics, pages 311-318.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput. Surv. 55(12).
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9.
Pei Ke, Haozhe Ji, Yu Ran, Xin Cui, Liwei Wang, Lin- feng Song, Xiaoyan Zhu, and Minlie Huang. 2021. JointGT: Graph-text joint representation learning for text generation from knowledge graphs. In Find- ings of the Association for Computational Linguis- tics: ACL-IJCNLP 2021, pages 2526-2533 Online. Association for Computational Linguisties.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485 5551.
Moniba Keymanesh, Adrian Benton, and Mark Dredze. 2022. What makes data-to-text generation hard for pretrained language models? In Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics ( (GEM), pages 539-554, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Clement Rebuffel, Thomas Scialom, Laure Soulier, Ben- jamin Piwowarski, Sylvain Lamprier, Jacopo Staiano. Geoffrey Scoutheeten, and Patrick Gallinari. 2021. Data-questeval: A referenceless metric for data-to- text semantic evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 8029-8036.
Klaus Krippendorff. 2018 Content analysis: An intro- duction to its methodology. Sage publications.
Leonardo F. R. Ribeiro, Martin Schmitt, Hinrich Schütze, and Iryna Gurevych. 2021. Investigating pretrained language models for graph-to-tex genera- tion. In Proceedings of the 3rd Workshop on Natural
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.

---

Language Processing for Conversational AA, pages 211-227, Online. Association for Computational Lin- guistics.
Victor Zhong. Caiming Xiong, and Richard Socher, 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv: 170.00033..
Leonardo F. R. Ribeiro, Yue Zhang, Claire Gardent, and Tryna Gurevych. 2020. Modeling global and local node contexts for text generation from knowledge graphs. Transactions of the Association for Compu- tational Linguistics, 8:589-604.
Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. 2018. Commonsense knowledge aware conversation generation with graph attention. In UCAI, pages 4623 4629..
Julien Romero and Simon Razniewski. 2020. Inside quasimodo: Exploring construction and usage of commonsense knowledge. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages
A
Appendix
A.1 Metrics for the evaluation of G2T
BLEU (Papineni et al., 2002) This is one of the oldest sentence similarity metrics. BLEU is computed weighted geometric mean of n-gram precision scores, that is: rewards inputs that share many common substrings (or n-grams, usually n is 4). The score ranges between and 1.
Ananya Sai, Akash Kumar Mohankumar, and Mitesh M Khapra. 2022. A survey of evaluation met- ries used for nlg systems, ACM Computing Surveys (CSUR), 55(2):1-39.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metries for text generation. arXiv preprint arXiv:2004. 04696.
METEOR (Banerjee and Lavie, 2005) This similarity metric is based on the harmonic mean of the 1-gram precision and recall. Its recall has higher weight than precision. METEOR scores range between and 1.
Anastasia Shimorina, Claire Gardent, Shashi Narayan, and Laura Perez-Beltrachin.. 2018. WebNLG chal- lenge: Human evaluation results. Ph.D. thesis, Loria Inria Grand Est.
Prajna Upadhyay. Oana Balalau, and Ioana Manolescu, 2023. Open information extraction with entity fo- cused constraints. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1285- 1296, Dubrovnik, Croatia. Association for Computa- tional Linguistics.
BERTScore (Zhang et al., 2019) This leverages the pre-trained contextual embeddings of BERT, and matches words in candidate and reference sen- tences by cosine similarity. The score can be used to compute precision, recall, and F1. BERTScore is also a similarity measure, ranging from to 1.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
BLEURT (Sellam et al., 2020) This is fine- tuned Bert model on synthetically generated texts. It provides a similarity score learned from evalu- ation scores such as BLEU or ROUGE, and from human ratings; 1 represents perfect match, while a value close to 0 means no similarity. We consider the authors' latest released model, BleuRT-20
Rand Wilcox. 2016. Comparing dependent robust correlations. British Journal of Mathematical and Statistical Psychology, 2953)2215-224.
Wen-tau Yih, Ming- Wei Chang, Xiaodong He, and Jian- feng Gao. 2015. Semantic parsing via staged query graph generation: Question answering with knowl- edge base. In Proceedings of the 53rd Annual Meet- ing af the Association for Computational Linguistics and the 7th International Joini Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 1321-1331, Beijing. China. Association for Computational Linguistics.
BARTScore (Yuan et al., 2021) This metric is based on the BART seq2seq language model: the score represents the weighted log probability of one text y. given another text æ. We compute two versions of the BARTScore, based on the indica- tions of the paper: in the first version of the score, we compare the generated sentence with the gold standard (we refer to it as BARTScore, the default version provided by the authors), and second version in which we compare the generated text with the input graph (referred in the paper as the faithfulness method, BARTScore-faithie in our ex- periments). The second score should encourage
Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text text gener- ation. Advances in Neural Information Processing Systems, 34:27263-27277..
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019, Bertscore: Eval- uating text generation with bert. arXiv preprint arXiv:1904.0965..
hittps://githin comm/gooleeresserch/bbaaaae

---

the metric consider the factual faithfulness of generation. BARTScore is not normalized.
ii) If the confidence intervals of M1, M2 overlap. we need check whether a statistically significant conclusion can drawn about the metrics. This is the case only if the confidence interval of the dif- ference in correlation between the metrics is above 0 (Wilcox, 2016), in which the metric with higher mean correlates better with human judgment.
A.2 Bootstrapping
Given two metries M1, M2, we need to assess which one better correlates with human annota- tions. We have at our disposal set of G2T systems ,i. and set A of G2T samples. Given set D of dimensions, e.g., fluency, factfulness, etc., we have our disposal the human annota- tion (S)))) d E D 1 S i S,a E A: this is the score that human users assigned to the G2T generation of system S,, on the sample a, along dimension d. Similarly, each automated metric M computes (a))) the M score of S; on sam- ple a along dimension d. We note however that automated metric give generally just one score for all dimensions.
A.3 Improved Samples of G2T Generation
Table 9 shows the improvement of factual faith- fulness in G2T generation task after integrating FactSpotter into G2T inference. The samples are from the hardest zero-shot and compositional splits of GrailQA dataset. Each sample in the table in- cludes a description of a subgraph, a ground truth text, text generated by the best baseline model (JointGT), and a text generated by our FactJointGT, Although the texts generated by the baseline model, JointGT, are generally fluent, we observe hallucinations in its generations: i) Some predi- cates are interpreted with different meanings in the generated texts; ii) Some entities are generated wrongly; ii) Some facts in subgraphs are lost in the generated texts. After integrating FactSpot ter into the inference of G2T generation, without retraining the G2T model, our FactJointGT can generate texts that verbalize facts in subgraphs more correctly and completely. Hence the factual faithfulness of G2T task improved by FactSpotter, but the other metrics do not necessary improve, since rephrased high quality texts might also be punished.
Based on the (a))). (s(a(()) over all systems, samples, and dimensions, we assess how well metric M correlates with human judgment H in statistically sound way. Since each dimension is of independent interest, we study the correlation of M and II separately for each dimension, and omit d from the notations below, for simplicity.
We compute the three correlation coefficients (r, P and r) between vectors of per-system scores (M))., (Ss)) and [H))),.. ... (Ss)) One aspect remains to settle: for a given S;, how aggregate information from all the annotated samples A? It important to do this in a way that is robust to the divergence and noise sometimes found in human annotations.
A.4 Parameter tuning
A.4.1 Percentages in negative samples for FactSpotter training
As is proposed in (Dhingra al., 2019), apply bootstrapping, as follows. We generate a number (we used 1000, which same as (Dhingra et al., 2019)) of "variants" A of the an- notated samples A, by sampling with replacement from A. Each A has the same size as A, but it may contain some A samples several times, and others not at all. We then compute correlation coef- ficients between the vectors (Mi (, M (Ss) and (S))..... ... (Ss) over each A,, then repre- sent each of the resulting 1000 correlation values by their mean, together a confidence interval. The relatively large number of "variants" acts as insula- tion against divergence and noise in the annotations. Then for two metrics M1, M2:
FactSpotter receives in the training set two types of negative samples. In the first type of negative sample (type I) we modify the fact from the graph, while the second type of negative sample (type we modify the text. To determine the best balance between negative samples of type respectively, type II, we compute the correlations between the score of FactSpotter and a subset of 50 annotations per system from the 2020 WebNLG challenge. As shown in Table 10, the best results for detecting factual accuracy and fluency are using either ratio 0.9-0.1 0.8-0.2. In the remaining of the paper, FactSpotter with the parameters 0.9-0.1.
i) If the mean of Mj is higher than M2 and their confidence intervals do not overlap, means M1 better correlates with human judgment, than M2.
A.4.2 À tuning for factual generations
We recall that in Equation 1, we have defined a weight À that quantifies the importance that we

---

Subgraph (spacecraft, manufacturer, International Telephone & Telegraph).
Ground Truth What is the total number of spacecrafts manufactured by International
JointGT Baseline
FactJointGT
International Telephone & Telegraph
How many spaccerafts did International
Telephone & Telegraph manufacture with Boeing Company?
(spacecraft, manufacturer, Boeing Company), (spacecraft, count, (music recording, artist, Celine Dion),
manufactures how many spacecrafts?
Telephone & Telegraph and Boeing Company?
What is the musical recording that was recorded by Celine Dion? How many video
Which musical
What is the shortest Celine Dion song?
recording was recorded by Celine Dion with the shortest length? How many video game expansions were published by
(music recording, length, float), (float, argmin, (John Elliott, games published, computer game expansion), (computer game expansion, count,) ) (conference series, conferences of this type,
How many game expansions has John Elliott released?
game expansions
were published by John Ellison?
John Elliott?
What conference series that has sponsoring organization of Australian Centre for Psychoanalysis and is of type seminar?
What conference
Australian Centre for Psychoanalysis sponsors which conference series?
(ominar))(confereee series, sponsoring organization, Australian Centre for Psychoanalysis), (conference series, question) (railway, terminuses, Strabanc), (railway, question)
series is sponsored by Australian Centre for Psychoanalysis and is a seminar?
Strabane is the terminuses of which railway?
Which railway has a terminus of Stratane?
Strabane is the terminus of which railway?
Table 9: Texts with improved factual faithfulness after integrating FactSpotter ter into G2T inference. Red stands for the contents that are presented in subgraphs, but missed in the baseline generations. Blue stands for the exact contents correctly presented in ground-truth texts or FactJointGT generations.
Type T/
Data Coverage
Relevancel
Finency
of X on the different splits of GrailQA on the T5 small model. The BleuRT score does not vary sig- nificantly, meaning that the fluency of the G2T model should not be affected. We fix À to 0.15 for the experimental evaluation in Section 6, as it gives the best performance on the zero shot split, which is the most challenging, and gives very good results also for the other splits. More precisely, we report the results of FactT55 and FactJointGT with 0.15 on the datasets introduced in Section 6.3.
Type II
0.9/0.1
0.85
0.84
0.68
0.93
0.88
0.75
0.91
0.79
0.63
0.8/0.2
0.83
0,92
0.93
0.79
0.61
0.7/0.3
0.77
0.84
0.66
0.85
0.89
0.73
0.66
0.63
0.50
Table 10: The influence of the ratio on FactSpotter.
HD
Zero-shot BleuRT
Compositional BleuRT
A
BleuRT
Facts
Facts
Facts
0.0
69.53
97.98
62.27
93.27
63.99
94,94
0.05
69.76
98.15
62.94
95.15
63.92
95.69
0.1
69.76
98.48
63.00
94.56
63.71
95.83
0.15
69.63
98.47
63.01
94.60
63.72
96.58
A.5
Sentence level correlation
0.2
69.47
98.38
62.95
94.79
63.61
96.19
We explain two distinct definitions of sentence- level correlation provided by literature and report the correlations on the WebNLG human annota- tions.
Table 11: BleuRT vs. FactSpotter when varying the parameter (importance of the fact classifier for generat- ing sentence, see Equation l).
The first definition of sentence-level correla- tion (Colombo et al., 2022) is outlined as follows:
assign the probability that the next word is pre- dicted such that increase the probability of hav-
Construct pair containing the automated metric scores [M(sys Msyss))] and the corresponding system-level human scores [H (sys)), (syss)] for a given sentence.
ing a fact in the generated text, versus the default probability of choosing a word based on previously generated tokens. In Table 1 1,, we vary the value

---

The Pearson correlation amidst these pairs is computed. If the correlation is significant (adopting p < 0.05 as the threshold), this correlation is preserved.
Metrics
Sem. Adeq.
T. Structure
Fluency
Human
0.80
0.77
0.77
BertF1
0.80
0.78
0.75
BleuRT
0.82
0.79
0.78
BartS
0.81
0.77
0.78
Then, report the average over significant correlations (average over at most the total number of annotated sentences).
FactS
0.84
0.78
0.78
Table 12: The result of sentence-level correlation with the first definition the WebNLG 2017 annotation.
When there is a large number of significant pairs with high correlation value, it can show if metric can be used to compare different verbal- isations of the same imput triples. We report the human correlation results computed this def- inition on the WebNLG 2017 and 2020 annota- tions in Tables 12 and 13 respectively. We note that only FactSpotter is significantly larger than the other metrics on data coverage and relevance for WebNLG 2020, and semantic adequacy for WebNLG 2017, which are metrics about factual faithfulness. Please note that here it is important to report how many sentences the correlation was significant with p 0.05. Because we compute if metric has higher score than second metric using the bootstrapping technique presented in the An- nexes, different samples might have different num- ber of sentences with correlation having p < 0.05, hence we report the average number of sentences over all the samples. For WebNLG2017, out of 223 human annotated sentences, have for semanties:
Metries
Correct.
D. Cover.
Fluency
Relev.
T: Struct,
Human
0.67
0.68
0.62
0.65
0.67
BerFI
0.65
0.67
0.64
0,66
0.66
BleuRT
0.72
0.68
0.68
0.71
0.70
BartS
0.71
0.69
0.66
0.70
FactS
0.69
0.50
0.71
0.60
Table 13: The result of sentence-level correlation with the first definition on the WebNLG 2020 annotation.
Metrics
Sem. Adeq.
T. Structure
Fluency
Human
0.56
0.44
0.50
BertF1
0.64
0.54
0.57
BleuRT
0.69
0.55
0.59
BartS
0.59
0.47
0.45
FactS
0.65
0.43
0.43
Table 14: The result of sentence-level correlation with the second definition on the WebNLG 2017 annotation.
Currect.
Cover.
Fluency
Structure
Human
0.38
0,29
0.35
0.30
BerFF
0.42
0.42
BleuRT
0.45
0.47
0.38
0.43
BLEURT 171, FactSpotter 172, BERT 121 and BART 169. For fluency we have BLEURT 80 sen- tence pairs, FactSpotter 48, BERT 71 and BART 55. For textual structure BLEURT had 104 sentences pairs, FactSpotter 65, BERT 94 and BART 75. We note that for semantics the majority of pairs are sig- nificant (172 out of 223), meaning that FactSpotter can predict with high correlation improved verbal- isations of the same input graph, hence it can be used to solve Q2. For WcbNL2017, out of 178 human annotated sentences, we have: for correct- ness, BLEURT 121, FactSpotter 83, BERT 95 and BART 97 significant pairs on average; ii) for data coverage, BLEURT 102, FactSpotter 100, BERT 65 and BART 106; üii) for fluency, BLEURT 105, FactSpotter 49, BERT 90 and BART 72; iv) for relevance, BLEURT 99, FactSpotter 67, BERT 71 and BART 78; v) for text structure, BLEURT 92, FactSpotter 42, BERT 75 and BART 59; In this dataset, we obtain far less significant pairs, but we will show next that the annotators had more dis- agreement on WebNLG 2020.
BanSS
0.39
0.38
0.32
0.35
0.31
FactS
0.38
0.40
0.29
0.37
0,27
Table 15: The result of sentence-level correlation with
the second definition on the WebNLG 2020 annotation.
The second definition of sentence-level corre- lation (Banerjee and Lavie, 2005) is computed be- tween the vector containing all the automatic scores for each sentence by system S given by metric M, and the vector containing the human metrics for each sentence. The sentence-level correlation of a metric M is computed as an average over all the correlation of each system Si,..., Ss. This measure determines a metric can be used to rank verbalisations of different input graphs. The human correlation results computed as the second defini- tion on the WebNLG 2017 7 and 2020 annotations reported in Tables 14 and 15 respectively.
On WebNLG 2017 dataset, three metrics achieve similar scores for semantic adequacy, with no re- sults significantly larger than the others (computed

---

Metric
Krippendorff's Alpha
not improve agreement. In the 2017 challenge, the annotators gave scores from to 3, but we believe that those scores were not sufficienlly described such that the annotators could choose them accord- ingly. We recommend that future annotations use a Likert scale annotation, but with clearer guidelines and examples for each score.
Fluency 0.4201
T. Struct.
0.3231
Sem. Adeq.
0.5314
Table 16: Krippendorff's alpha for ordinal metrics on WebNLG 2017 human annotations
Metric
Krippendorff's Alpha
A.6 Qualitative Analysis for examples with different number triples
Correct.
0.2769
D. Cover.
0.2632
Fluency 0.2580
We conducted a qualitative analysis on examples with different number of triples, to observe where the proposed metric shows improvements during decoding. We present the number of (graph, text) pairs as categorized by the number of triples within the input graph for the test set, and results for vari- ous datasets segmented by the number of triples are reported below. SimpleQuestion was not consid- ered because it has only one triple per input graph. For the WebNLG 2017 dataset, that for the small G2T model, T5S, the triples of size 4, and are more challenging. as is shown in Table 19. The integration of FactSpotter enhances the factuality of the results (as evident in F-T5S). We also observed a distinct improvement across various numbers of triples in the WebNLG 2020 dataset, especially when handling sentences com- prising multiple triples.
Relev.
0.1860
T. Struct.
0.2234
Table 17: Krippendorff's alpha for ordinal metrics on WebNLG 2020 human annotations
using confidence intervals by bootstrapping tech- nique Appendix A.2). Using (Evans, 1996) cor- relation guidelines, where a value between 0.6 to 0.79 is "strong correlation" and 0.8 to 1.0 is "very strong", we have "strong" correlation for the type of sentence correlation to the annotation. However, on WebNLG 2020 dataset, all metrics demonstrate "moderate" level of correlation, given that the scores hover between 0.3 and 0.49.
observe
We note that the two sentence correlation inter- pretations presented above are more sensitive to noise in the human annotation, as we consider each individual sentence score, not aggregates. This is evident also from the lower Hluman-Human cor- relation observe sentences, in comparison with the high correlation for the system level cor- relation. In Tables 16 and 17, we present the inter- annotator agreement for WebNLG 2017 and 2020 using Krippendorff's alpha (Krippendorff, 2018). It can be observed that both datasets exhibit gen-
As for the DART dataset in Table 23, inputs ranging from to 5 triples witness improved re- sults with FactSpotter's 's addition. Remarkably high SactSpotter scores are observed for inputs contain- ing 6 or 7 triples. It's worth noting that of the inputs with 6 triples, 569 out of 598 are sourced from the E2E split, and for those with 7 triples, 320 out of 342 hail from the same E2E split. Hence, the E2E split DART seems to be less challenging.
erally low consistency among annotators, with the 2020 dataset showing less agreement than its 2017 version. However, on WebNLG 2017 we observe a higher agreement on semantic adequancy, for which we also observed high correlation with our metric at sentence level.
The results of GrailQA dataset are illustrated in Tables 25, 29, and 27. Regarding the GrailQA zero-shot split, single-triple verbalization consis- tently achieves superior scores with tactSpot Beginning with two triples, the incorporation FactSpotter offers a discernible boost in model performance. In the compositional split, mirror- ing the trend observed in the zero-shot scenario, FactSpotter scores remain high for single-triple in- puts. Furthermore, there marked improvement for inputs with two more triples upon the addi- tion of FactSpotter. In the IID split, while the base- line scores are impressively high, enhancements is noticeable for small model like T5S.
We believe that these results show the need for better human annotation guidelines for this task. In particular, on WebNLG2020, the annotators were asked to give score from 100 to a sentence for given dimension such as correctness. Such fine-grained decision is very difficult to take, hence the low agreement score. We note that mapping these scores to scores from to or to does

---

Triple Num
6
Triple Num
2
3
4
Pair Num
1540
821
419
241
185
0
Pair Num
1231
522
55
20
Percentage
47.95
25.56
13.04
7.50
5.76
0
0.19
Percentage
67.34
28.55
3.00
1.09
Table 18: The number and the percentage of (graph, sentence) pairs for the test set of WehNLG 2017 Const
Table 24: The number and the percentage of (graph, sentence) pairs for GrailQA zero-shot split
Triple Num
2
4
7
T5S
99.87
99.45
99.13
97.62
6.66...
98.09
97.53
Triple Num
2
3
4
F-T5S
99.75
99.65
99.22
98.74
99.20
99.93
TSS
96.41
89.63
92.71
93.62
TSB
99.65
99.69
99.35
99.15
99.52
99.93
F-TSS
97.08
91.30
94.52
99.39
F-T5B
99.81
99.70
99.69
99.85
99.53
99.93
JGT-T5
99.69
99.73
98.52
98.63
99.30
99.93
T5B
96.49
93.03
92.84
94.98
FGT-T5
99.40
99.89
99.50
99.89
99.60
99.93
F-T5B
97.48
93.31
94.28
99.68
JGT-T5
96.96
90.77
93.50
91.28
Table 19: FactSpotter by the number of triples for the test set of WebNLG 2017 Const
FGT-TS
98.02
91.63
95.95
93.81
Table 25: FactSpotter by the number of triples for for GrailQA Zero-shot split
Triple Num
2
5
Pair Num
369
698
1050
1230
1065
684
553
Percentage
9.09
17.19
25.86
30.04
26.21
16.84
13.62
Triple Num
2
3
4
Table 20: The number and the percentage of (graph,
Pair Num
388
292
39
38
sentence) pairs for the test set of WebNLG 2020
Percentage
51.25
38.57
5.15
5.01
Triple Num
6
T5S
93.71
93.91
92.24
90.71
89.95
88.68
86.12
Table 26: The number and the percentage of (graph. sentence) pairs for GrailQA Compositional split
F-T5S
95.05
95.13
95.58
94.07
92.21
91.01
TSB
95.21
96.33
94.51
93.00
92.29
92.30
93.36
F-T3B
96.27
96.74
95.69
94.41
94.72
94.89
94.09
JGT-T5
94.74
94.71
92.71
92.16
88.16
89.10
88.46
Triple Num
2
3
4
FGT-TS
94.26
95.85
93.85
93.93
90.90
91.43
92.48
TSS
98.47
94.89
95.53
88.15 15
F-TSS
99.14
95.77
98.12
94.69
Table 21: FactSpotter by the triple number for the test set of WebNLG 2020
T5B
98.64
96.23
77.78
93.27
F-T5B
99.25
96.35
93.80
94.36
Triple Num
2
6
JGT-T5
98.84
95.59
89.73
91.02
Num
797
342
FGT-T5
98.92
96.62
98.79
94.85
Percentage
16.64
15.64
16.11
17.05
16.13
11.73
6.71
Table 22: The number and the percentage of (graph,
Table 27:
FactSpotter by the number of triples for GrailQA Compositional split
sentence) pairs for DART test
Triple Num
4
6
Triple Num
2
3
4
T5S
98.47
93.55
93:35
92.45
94.01
99.57
99.82
Pair Num
257
465
63
12
F-T5S
99.71
97.21
96.41
94.82
96.68
99.88
99.82
TSB
98.08
95.53
94.54
94.44
96.03
99.88
99.99
Percentage
32.65
58.34
7.90
1.50
99.54
96.32
96.16
97.19
99.74
JGT-T5
98.49
94.36
93.61
93.44
94.33
99.82
99.80
Table 28: The number percentage of (graph, sentence) pairs for GrailQA IID split
FGT-T5
99.51
96.43
95.83
94.78
95.82
99.77
99.89
Table 23: FactSpotter by triple number for DART
Triple Num
3
4
T5S
97.75
97.98
98.82
96.96
A.7 Further analysis on the difficulty of graph-to-text generation
F-T5S
97.96
98.73
98.83
99.20
T5B
99.55
99.31
99.38
99.77
We consider the intricacy in verbalizing a given triple is predominantly influenced by:
F-T5B
99.55
99.47
99.38
99.77
JGT-T5
98.09
99.12
98.31
99.77
FGT-T5
98.72
99.35
99.31
99.77
1. The distance between entity names in the knowledge graph and those in natural text.
Table 29: FactSpotter by the number of triples for
GrailQA IID split
2. The difference between KG predicates and their equivalent natural language phrases.

---

Entity Generation Regarding the difficulty of generating correct entities, we have the following statistics on the WebNLG 2017 dataset (v2.1):
Summary of Difficulty Datasets like WebNLG (from DBPedia) and GrailQA (from FreeBase) present challenges on different fronts. The dif- ficulty from datasets based on FreeBase is the distance between knowledge graph and natural language is much higher. However, WebNLG and DART datasets have more complex input sub- graphs, which has more number of triples, while GrailQA only has input subgraphs with up to 4 triples.
87% of input graph entities appear identical in its corresponding ground-truth texts.
7% of differences arise from special charac- ters, c.g., "Motherwell F.C." in the input graph becomes "Motherwell FC" in the ground- truth, both being correct.
We consider that promoting accurate predicate generation is more challenging problem than pro- moting the generation correct entities, because predicates are much more often rephrased in sen- tences, which is harder to evaluate. Inspired by the efficacy of Constrained Beam Search in ensuring accurate entity generation, we designed FactSpotter to enhance the accurate production of rephrased facts, especially rephrased predicates to be cor- rectly generated.
2% due to variations in date number formats, observed with "2006-124" 12-31" in the KG interpreted to be "December 31, 2006" in the ground-truth text.
1% emerge from linguistic differences be- tween the source language and English, as with "Atatürk" in KG being written as "Ataturk" in reference text.
Other variances often regard alternative en- tity designations, such as "United States" in the KG being abbreviated as "US" in specific ground-truth texts.
For the GrailQA dataset, rooted in Freebase, 99% of input graphs retain consistent entity names their ground-truth sentences. Accurate entity verbalization relatively straightforward for this dataset, but we observe hallucinations in the texts generated by baseline models.
Considering that most differences between KG and natural language entity names result from for- matting nuances, we can easily address its con- sistency challenge using constrained beam search ensure all entity names appear in the generated text. For beam sizes over 10, the text generation becomes accurate without compromising other met- rics. However, for beam sizes under 10, such con- straints tend to impede the fluency.
Predicate Generation In the WebNLG 2017 dataset, 49% of predicates appear the same as the KB in their corresponding ground-truth sentences. Using SBerr, we computed an average similarity of 84% between each predicate and its nearest n- grams in the ground-truth.
For the GrailQA dataset, only 27% of predicates align perfectly with their KG representations in the ground-truth. The average similarity stands at 64%, highlighting a larger difference between FreeBase predicates and their natural language phrases.


Reviews:
[{"review_id": "EIDJE1IYV9", "reviewer": "Reviewer_v9ZD", "summary": "", "questions": "A: Line 273: How exactly is the FactSpotter score computed? Is it a sum of the binary 1, 0 prediction for each fact, or is it the sum of their probabablities (i.e. values after the softmax)? If it is the latter, won't your metric not be able to distinguish between a text where FactSpotter is moderately confident that all the facts are expressed in the text (i.e. 0.5 for each fact) versus another text where it is confident that half of the facts are not expressed in it?\n\nB: Section 6 - Do you retrain specific FactSpotter models for your generation experiments in this section, i.e. using the training data for each of the datasets you evaluate against (SimpleQuestions, GrailQA, WebNLG an DART)? Or do you reuse the FactSpotter model trained on WebNLG 2020 data in Section 5 for all these other datasets? If the latter case, the WebNLG 2017 test sets were incorporated into the WebNLG 2020 training data (https://synalp.gitlabpages.inria.fr/webnlg-challenge/challenge_2020/#data), i.e. this means that despite your use of the constrained split of the 2017 WebNLG test set, the FactSpotter would have seen all of the fact triples in the constrained test set. If so, this could explain why you achieve the highest automatics scores for the WebNLG test set (i.e. the FactSpotter model is leaking knowledge of the test set into the generation). In any case, your conclusion (Line 531) --- that WebNLG (or any dataset for that matter) is no longer a challenging dataset --- will benefit from more supporting analysis/evidence than merely based on it having the highest automatic scores obtained. \n\nC: Line 549: select the \"best FGT-T5 model...\", best with respect to what, or based on what measure?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reviewer v9ZD\n\nWe thank the reviewer for the feedback and suggestions! Below, we reiterate the reviewer's points and present our answers. \n\n### Questions For The Authors:\n\n*A: Line 273: How exactly is the FactSpotter score computed? Is it a sum of the binary 1, 0 prediction for each fact, or is it the sum of their probabablities (i.e. values after the softmax)? If it is the latter, won't your metric not be able to distinguish between a text where FactSpotter is moderately confident that all the facts are expressed in the text (i.e. 0.5 for each fact) versus another text where it is confident that half of the facts are not expressed in it?*\n\n \n**Answer:** Indeed, using binary results allows distinguishing sentences where FactSpotter gives 0.5 probability for all facts to exist, from sentences that lack half of the facts. While we only report probability averages, we implemented binary and probability versions of FactSpotter. We reported averaged probabilities to capture different positive results qualities, e.g., a generation with 0.9 probability may be more factual than another with 0.6 probability. We can provide both results in the final version of the paper, and in our publicly available code, we will also provide both options. Note that users can see both the score per fact (what is the probability that a triple is present in the sentence) and the average over all facts, making our score interpretable. In addition, when we use FactSpotter to improve the prediction of G2T models, we check the score per fact until it becomes positive. Hence we can distinguish a missing fact, from a fact with a moderate probability.  \n\n\n*B: Section 6 - Do you retrain specific FactSpotter models for your generation experiments in this section, i.e. using the training data for each of the datasets you evaluate against (SimpleQuestions, GrailQA, WebNLG an DART)? Or do you reuse the FactSpotter model trained on WebNLG 2020 data in Section 5 for all these other datasets? If the latter case, the WebNLG 2017 test sets were incorporated into the WebNLG 2020 training data, i.e. this means that despite your use of the constrained split of the 2017 WebNLG test set, the FactSpotter would have seen all of the fact triples in the constrained test set. If so, this could explain why you achieve the highest automatics scores for the WebNLG test set (i.e. the FactSpotter model is leaking knowledge of the test set into the generation). In any case, your conclusion (Line 531) --- that WebNLG (or any dataset for that matter) is no longer a challenging dataset --- will benefit from more supporting analysis/evidence than merely based on it having the highest automatic scores obtained.*\n\n\n**Answer:** We were also concerned about leaking knowledge from the test set during training. Therefore, **we retrain FactSpotter for each dataset, using only the corresponding training split**. \nWe have reported the results of the training in Table 1. We trained FactSpotter on the 2.1 version of WebNLG, which has the same split as WebNLG2017. \nFor the 2020 WebNLG human annotations, we kept the 2017 model, which has seen less data than WebNLG 2020. We have trained FactSpotter on WebNLG2020 (version 3 of the WebNLG data) and we show the changes that we get in respect to the paper:\n| Metric  | Correct. | D. Cover. | Relev. | \n| ------- | -------- | --------- | ------ | \n| FactSpotter WebNLG2020 |    0.94      |    0.93      |    0.95 |\n|  FactSpotter WebNLG2017 (paper)     |   0.94      |   0.91        |   0.96     | \n\nIn Table 5 and the evaluation of Section 6.4 and 7, we analysed WebNLG v2.1, with FactSpotter only trained on WebNLG V2.1. A different analysis is required for the version 3 of the dataset, WebNLG2020. We will make this clear in the final version of the paper.\n\n\n\nWe agree with your concern that we cannot conclude that WebNLG is no longer challenging based on it having the highest automatic score; line 531 was related to the evaluation we perform in Section 7. We will rewrite that paragraph to reflect only the information contained in Table 5, as follows: \"_Table 5 has the highest FactSpotter score from all datasets, which means that we observe the most factual generations on WebNLG, according to FactSpotter. We will perform a human evaluation of factufulness in WebNLG in the next section._\" \n\n\nRegarding the comment on the evaluation of challenging datasets, we add the following observations which we will also add to the paper. \n\nThe difficulty for verbalizing a given triple strongly depends on: 1) the distance of the **entity names** in the KG to the entity names in the natural language, 2) the distance from the **predicates** in KG, to the phrases in natural language. \n\n\nRegarding the difficulty of **generating correct entity names**, we have the following statistics on the **WebNLG** dataset (v2.1):\n- 87% of input graphs' entities appear identical in its corresponding ground-truth texts.\n- 7% differences are from special letters, such as commas, dots, quotation marks, and parentheses. For instance, “Motherwell F.C.” in input graph is given as “Motherwell FC” in ground-truth text. They’re both correct spellings.\n- 2% are from different date or number format, such as “2006-12-31” in knowledge graph is interpreted as “December 31 2006” in text.\n- 1% are from the difference between its original language and English. E.g., “Atatürk” from knowledge graph is written as “Ataturk” in golden text. Both entity names should be correct.\n- Other differences are same entity with different quotations, such as “United States” in KG is written as “US” in some ground-truth texts.\n\nIn **GrailQA**, a dataset based on Freebase, 99% input graphs have exactly same entity names in ground-truth sentences. Then verbalising the entities correctly for this dataset is a easy task. \n\nSince the distance between entity names in KG and natural language are mostly from formating, which all contain correct information, the problem of entity name consistency can be easily solved by using constrained beam search to force all entity names to appear in the generated text (When beam size is over 10, it can generate sentences with correct entities without suppressing other metrics. When beam size is less than 10, the fluency of generation decreases after adding constraints). We had investigates this line of research in our early experiments, however **constrained beam search does not easily generalize to enforcing predicates, because predicates are usually rephrased** in natural language. \n\nFor the difficulty of **generating correct predicates**, in WebNLG dataset, 49% of predicates already appear the same as in the KB in their corresponding ground-truth sentences. We compute the average similarity between each predicate and the most similar n-grams in ground-truth sentence, according to SBert. The average similarity is 84%.\n\nIn GrailQA dataset, based on Freebase, only 27% predicates appear exactly the same as in the KB in ground-truth sentences. The average similarity between each triple, and their most similar n-gram in the corresponding ground-truth text is 64%. So FreeBase predicates have a larger gap to their corresponding natural language phrases.\n\nWe note that the differences between datasets based on different KBs might be due to the underlying KBs or to the way the G2T datasets were created and predicates were chosen. \n\nThe WebNLG (from DBPedia) and GrailQA (from FreeBase) are difficult on different dimensions. The difficulty from datasets based on **FreeBase** is **the distance between knowledge graph and natural language is much higher**. However, WebNLG and DART datasets have **more complex input subgraphs**, which has more number of triples, while GrailQA only has input subgraphs with up to 4 triples.\n\nWe consider promoting correct predicate generation is a more challenging problem than promoting the generation of correct entities, because predicates are much more often rephrased in sentences, which is harder to evaluate. Inspired by Constrained Beam Search which could force all entities to be correctly generated, we train FactSpotter to promote rephrased facts, especially rephrased predicates to be correctly generated. \n\n\n*C: Line 549: select the \"best FGT-T5 model...\", best with respect to what, or based on what measure?*\n\n**Answer:**“best FGT-T5 model” is with respect to the FactSpotter metric. We will clarify this. \n\n\n### Reason to reject\n*While RQ1, RQ2 and RQ3 of the paper can be complementary, the method proposed/conclusions made by the authors in the sections on RQ2 and RQ3 hinges on RQ1 (i.e. their proposed metric). Although the proposed metric outperforms in the human evaluations correlations reported for RQ1 (Table 2 and 3 in Section 5), the correlation was computed on a **system level**. To use the proposed method from RQ1 to address RQ2 (i.e. as a source of training signal to improve G2T generation) and RQ3 (for assessing whether existing G2T datasets are no longer challenging), it is important to know the **sentence level correlation** with human judgements, in order to be able to support the claims and conclusions made by the authors for those RQs (Section 6 and 7). The paper also makes a number of broad sweeping statements which arguably are not sufficiently supported by their experimental set-up (see Questions to Authors).*\n\n**Answer:** We thank the reviewer for the opportunity to clarify. We base RQ2 and RQ3 not only on the human correlation results, but also on the high results in Table 1, in addition to the human evalution in Section 6.4 and 7. However, sentence-level correlation indeed strengthens our contribution and we can add it in the final version of the paper. The literature provides *two definitions of sentence level correlation*, both of which are interesting in our setting.  \n\nThe **first definition of sentence-level correlation** (from the paper \"InfoLM: A New Metric to Evaluate Summarization & Data2Text Generation\") is:\n - We construct a pair containing the automatic metrics scores [M(System_1), . . . M (System_S)] and the corresponding system-level human scores [H(System_1), . . . H(System_S)] for a sentence, and we compute the Pearson correlation between the pairs. If the correlation is significant (we take p<0.05), we save this correlation. \n - Then, we report the average over all significant correlations (average over at most the total number of annotated sentences).\n\nWhen we have a large number of significant pairs with high correlation value, this can show if a metric can be used to compare different verbalisations of the same input triples. This is important for Q2, where we use FactSpotter to improve the verbalisation of each individual sentence. \n\n\nWe report below the human correlation results computed as above on the WebNLG2017 annotation. These results support the findings on the 2.1 version of WebNLG that we used in sections 6.3,6.4 and 7.\n\n| Metric | Sem. Adeq. | T.Structure | Fluency |\n| ------ | ---------- | ----------- | ------- |\n| BertF1 |     0.80       |     0.78        |    0.75     |\n| BleuRT  | 0.82           |       0.79      |   0.78     |\n| BartS       |  0.81          |       0.77      |   0.78      |\n| FactSpotter |  0.84*          |    0.78         | 0.78       |\n\nWe note that only FactSpotter is significantly larger than the other metrics on semantic adequacy, the factufulness metric. Please note that here it is important to report on how many sentences the correlation was significant with p<0.05. Because we compute if a metric has a higher score than a second metric using the bootstrapping technique presented in the Annexes, different samples might have different number of sentences with correlation having p<0.05, hence we report the average number of sentences over all the samples. \nOut of 178 human annotated sentences, we have for semantics: BLEURT 171, FactSpotter 172, BERT 121 and BART 169. For fluency we have BLEURT 80 sentence pairs, FactSpotter 48, BERT 71 and BART 55. \nFor textual Structure BLEURT had 104 sentences pairs, FactSpotter 65, BERT 94 and BART 75. We note that for semantics the majority of pairs are significant (172 out of 178), meaning that FactSpotter can predict with high correlation improved verbalisations of the same input graph, hence it can be used to solve Q2. \n\nWe now move to the **second version of the sentence level-correlation**, (defined in paper METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments). This correlation is computed between the vector containing all the automatic scores for each sentence by a system S given by a metric M, and the vector containing the human metrics for each sentence. The sentence-level correlation of a metric M is computed as an average over all the correlation of each system S_1, ... S_S. This measure determines if a metric can be used to rank verbalisations of different input graphs. We report the results in the following table: \n\n| Metric | Sem. Adeq. | T.Structure | Fluency |\n| ------ | ---------- | ----------- | ------- |\n| BertF1 |     0.64       |     0.54        |    0.57     |\n| BleuRT  | 0.69           |       0.55      |   0.59     |\n| BartS       |  0.59         |       0.47      |   0.45      |\n| FactSpotter |  0.65          |    0.43         | 0.43      |\n\nWe observe that the three metrics achieve similar scores for semantic adequacy, with no results significantly larger than the others (computed using confidence intervals via the bootstrapping technique described in the Annexes). Using (Evans, 1996) correlation guidelines, where a value between 0.60 to 0.79 is strong correlation and 0.80 to 1.00 is very strong, we have strong correlation for the second type of sentence correlation. \n\nWe note that the two sentence correlation interpretations presented above might be more sensitive to noise in the human annotation, as we consider each individual sentence score, not aggregates. The inter-annotator aggrement on the human annotations was not reported in the original report of the WebNLG2017 data (WebNLG Challenge: Human Evaluation Results). \n\n### Missing references \n\nThank you for pointing out these references! \n\nIn **“Entity-Based Semantic Adequacy for Data-to-Text Generation”** the authors develop a metric for detecting if subgraph entities were generated in a verbalisation of the subgraph. Going beyond, our FactSpotter metric (a) also takes into consideration the predicate (not only the entities) and (b) it can be plugged in a G2T model. As highlighted above, enforcing entities in G2T is easier than enforcing predicates mentions. We can report also this metric on the top performing G2T models in the final version for completeness.  \n\nIn **“Data-QuestEval”**, the authors use a question generation (QG) and question answer (QA) framework, to evaluate the quality of generated text by testing whether each question from the QG module can be answered by the QA module. \nFactSpotter's goal is similar, but: (a) FactSpotter requires less computational resources as we do not need to use question generation and answering models, and the computational cost of a simple classifier for each fact is smaller than that of the sequence-to-sequence question answer model; (b) our model is self supervised, thus it does not requires additional data to the G2T model; ( c) our metric can be pluged into a G2T to improve its generation. \n\nWe will add this discussion in our paper. \n\n### Typos Grammar Style And Presentation Improvements\n\nWe thank the reviewer's suggestions to the writing of our paper. We'll improve it according to the suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Methods for automatically validating the factuality of text generated from graphs is important as G2T models become increasingly available and the barriers of access to them continue to decrease. It is important to have tools that can help detect whether these G2T generations (or any generated text in general) have poor factuality.", "reasons_to_reject": "While RQ1, RQ2 and RQ3 of the paper can be complementary, the method proposed/conclusions made by the authors in the sections on RQ2 and RQ3 hinges on RQ1 (i.e. their proposed metric). Although the proposed metric outperforms in the human evaluations correlations reported for RQ1 (Table 2 and 3 in Section 5), the correlation was computed on a **system** level. To use the proposed method from RQ1 to address RQ2 (i.e. as a source of training signal to improve G2T generation) and RQ3 (for assessing whether existing G2T datasets are no longer challenging), it is important to know the **sentence** level correlation with human judgements, in order to be able to support the claims and conclusions made by the authors for those RQs (Section 6 and 7). The paper also makes a number of broad sweeping statements which arguably are not sufficiently supported by their experimental set-up (see Questions to Authors).", "questions_to_authors": "A: Line 273: How exactly is the FactSpotter score computed? Is it a sum of the binary 1, 0 prediction for each fact, or is it the sum of their probabablities (i.e. values after the softmax)? If it is the latter, won't your metric not be able to distinguish between a text where FactSpotter is moderately confident that all the facts are expressed in the text (i.e. 0.5 for each fact) versus another text where it is confident that half of the facts are not expressed in it?\n\nB: Section 6 - Do you retrain specific FactSpotter models for your generation experiments in this section, i.e. using the training data for each of the datasets you evaluate against (SimpleQuestions, GrailQA, WebNLG an DART)? Or do you reuse the FactSpotter model trained on WebNLG 2020 data in Section 5 for all these other datasets? If the latter case, the WebNLG 2017 test sets were incorporated into the WebNLG 2020 training data (https://synalp.gitlabpages.inria.fr/webnlg-challenge/challenge_2020/#data), i.e. this means that despite your use of the constrained split of the 2017 WebNLG test set, the FactSpotter would have seen all of the fact triples in the constrained test set. If so, this could explain why you achieve the highest automatics scores for the WebNLG test set (i.e. the FactSpotter model is leaking knowledge of the test set into the generation). In any case, your conclusion (Line 531) --- that WebNLG (or any dataset for that matter) is no longer a challenging dataset --- will benefit from more supporting analysis/evidence than merely based on it having the highest automatic scores obtained. \n\nC: Line 549: select the \"best FGT-T5 model...\", best with respect to what, or based on what measure?", "paper_topic_and_main_contributions": "This work proposes a metric for establishing whether a knowledge base fact is correctly mentioned in a text. The method involves training a Electra (small) model with a binary classification objective on graphs and text from the training set of the WebNLG 2020 challenge. Evaluation was done by comparing the proposed metric's correlations (system-level) with human evaluation judgements from the 2017 and 2020 WebNLG graph-to-text (G2T) challenges. In addition, the authors propose applying their metric as a method to provide training signal when fine-tuning G2T models. They also claim that the results from their application of the metric to finetuning two model families (one based on pretrained language models with linearisation of the graphs, and another has a graph neural network component to model the  structure of the graph input) show that four existing G2T datasets, SimpleQuestions, GrailQA, WebNLG and DART are no longer challenging.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "- Entity-Based Semantic Adequacy for Data-to-Text Generation (Faille et al, 2021)\n- Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluation (Rebuffel et al, 2021)"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Line 153: it might be better to separately state that BLEURT is fine-tuned on human ratings that does consider semantic adequacy (or factual faithfulness), but not only that (i.e. fluency, grammaticality too). \n- Section 6.4 can benefit from better organisation, with headers and clearer writing to highlight what aspects of RQ2 is addressed. Currently it reads as a mix of a discusson of RQ2 and RQ3 (see Line 531), as well as a discussion of your error analysis of the generations. \n- Section 6.2 can benefit from clearer labeling each of the models and describing what they are and how they differ from each other. This way it will allow the reader to more easily understand and interpret the results reported in Table 4,5,6,7."}}, {"review_id": "mggE8w7m9p", "reviewer": "Reviewer_NiRu", "summary": "", "questions": "- I understand that the metric receives as input a triple and a text that contains the fact described in the triple but also describes other facts. Why don't the authors consider the whole graph as input instead of using a unique triple? Having a unique input for the graph would benefit the decoding?\n- Do you have some thoughts on why the T5 base models have worse performance on FactS compared to small models (Table 4)? I would expect that larger models would generate more faithful texts to the input graph.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for feedback and suggestions! Below, we reiterate the reviewer's points and present our answers. \n\n### Questions For The Authors:\n\n*1. I understand that the metric receives as input a triple and a text that contains the fact described in the triple but also describes other facts. Why don't the authors consider the whole graph as input instead of using a unique triple? Having a unique input for the graph would benefit the decoding?*\n\n**Answer:** We initially also considered promoting facts that can be inferred from the knowledge graph, by combining a reasoning model with our FactSpotter. However, we observed that inferred information is not always mentioned in the golden texts in datasets. For example, in WebNLG dataset, given input subgraph\n\n    Aaron_Bertram | associatedBand/associatedMusicalArtist | Kids_Imagine_Nation\n    Aaron_Bertram | associatedBand/associatedMusicalArtist | Suburban_Legends\n    Suburban_Legends | bandMember | Brian_Robertson_(trombonist)\n    Suburban_Legends | genre | Pop_music\n\nand ground truth text \n\n    Aaron Bertram plays for the Kids Imagine Nation band and the pop group Suburban Legends. \n    Brian Robertson is a member of the Suburban Legends and plays trombone.\n\nWe can infer that Aaron Bertram and Brian Robertson are colleagues from the knowledge graph, but in the ground-truth sentences, this relation is not mentioned. Hence, we decided to use FactSpotter to promote only explicit facts from the input. Since we already observed that many single facts from input subgraphs are not in the texts generated by baseline models, we consider this contribution is valuable. \n\nIn addition, the advantages of testing whether each triple is expressed well in a generated sentence are: \n(1) It makes the metric more **explainable**, since each atomic classifier can tell whether each triple is lost in a generated sentence or not; giving the whole graph as input to FactSpotter does not allow this. \n(2) It makes it easy to **embed FactSpotter into decoding** for more factual generation, as the reviewer mentioned. We can check whether each fact in each triple is already explained or not in the generation process, and then encourage the generation of each not included fact dynamically, according to Equation 1. For the facts that are already mentioned in the previously generated sequence, we no longer promote related information. Our goal is to _prevent generating repeatedly the same facts_, which is more useful than _always encouraging the generation of all facts in the graph_. \n\n\n*2. Do you have some thoughts on why the T5 base models have worse performance on FactS compared to small models (Table 4)? I would expect that larger models would generate more faithful texts to the input graph.*\n\n**Answer:** We believe that the larger models are **overfiting on SimpleQuestions**. The SimpleQuestions dataset is very large, but it has only 1-triple input graphs, many of which are about the same entities. The models might learn combinations of tokens during training, rather than interpreting knowledge graphs correctly. We have observed, for example, that \"Nintendo\" is often predicted as a game company by baseline models, even if the original input triple referred to another game company. \n\n### Reasons to reject\n\n*1. It would be good to see ablation studies and qualitative analysis for examples that the proposed metric improves during the decoding. For example, in examples that contains more facts (e.g., 5 triples) the model might have more difficulty generating factually generated text. Investigating such cases could bring insights and better illustrate the metric capabilities.*\n\n**Answer:** We thank the reviewer for the useful suggestion. Indeed the cases with more input triples are more challenging: this is where the best improvements are obtained by pluging  our metric in a G2T. We report the results per number of triples for the datasets below (with just 1 triple per input graph, SimpleQuestion is not considered). We will add these results to the paper.\n\n**WebNLG17 Const Test**\nThe number of  (graph, sentence) pairs, per number of triples in the input graph, in the test set: \n\n|Triple Num|1|2|3|4|5|6|7|\n|-|-|-|-|-|-|-|-|\n|Number of pairs|1540|821|419|241|185|0|6|\n|%age of pairs|47.95|25.56|13.04|7.50|5.76|0|0.19|\n\nIt happens that no graph consists of exactly 6 triples.\n\nFactSpotter by size of the graph (number of triples): \n\n|Triple Num|1|2|3|4|5|7|\n|-|-|-|-|-|-|-|\n|T5S|99.87|99.45|99.13|97.62|98.09|97.53|\n|F-T5S|99.75|99.65|99.22|98.74|99.20|99.93|\n|T5B|99.65|99.69|99.35|99.15|99.52|99.93|\n|F-T5B|99.81|99.70|99.69|99.85|99.53|99.93|\n|JGT-T5|99.69|99.73|98.52|98.63|99.30|99.93|\n|FGT-T5|99.40|99.89|99.50|99.89|99.60|99.93|\n\nFor WebNLG, we can observe that for the small model G2T, T5S, the triples of size 4, 5 and 7 are more challenging, and by pluging in FactSpotter we can improve their factfulness (F-T5S). \n\n\n**GrailQA**\n\n\nZero-shot FactS Result\n\n\n| Triple Num      | 1     | 2     | 3    | 4    |\n| --------------- | ----- | ----- | ---- | ---- |\n| Number of pairs | 1231  | 522  | 55  | 20    |\n| %age of pairs   | 67.34 | 28.55 | 3.00 | 1.09 |\n\n\nFactSpotter by size of the graph (number of triples):\n\n| Triple Num | 1     | 2     | 3     | 4     |   \n| ---------- | ----- | ----- | ----- | ----- |\n| T5S        | 96.41 | 89.63 | 92.71 | 93.62 |   \n| F-T5S      | 97.08 | 91.30 | 94.52 | 99.39 |   \n| T5B        | 96.49 | 93.03 | 92.84 | 94.98 |   \n| F-T5B      | 97.48 | 93.31 | 94.28 | 99.68 |   \n| JGT-T5     | 96.96 | 90.77 | 93.50 | 91.28 |   \n| FGT-T5     | 98.02 | 91.63 | 95.95 | 93.81 |   \n\nOverall, 1-triple verbalisations receive higher FactSpotter score; from 2 triples on, plugging FactSpotter improves the model performance. \n\n\nCompositional FactS Result\n\n| Triple Num      | 1     | 2     | 3    | 4    |\n| --------------- | ----- | ----- | ---- | ---- |\n| Number of pairs | 388  | 292  | 39  | 38    |\n| %age of pairs   | 51.25 | 38.57 | 5.15 | 5.01 |\n\nFactSpotter by size of the graph (number of triples):\n|Triple Num|1|2|3|4|\n| --------------- | ----- | ----- | ---- | ---- |\n|T5S|98.47|94.89|95.53|88.15|\n|F-T5S|99.14|95.77|98.12|94.69|\n|T5B|98.64|96.23|77.78|93.27|\n|F-T5B|99.25|96.35|93.80|94.36|\n|JGT-T5|98.84|95.59|89.73|91.02|\n|FGT-T5|98.92|96.62|98.79|94.85|\n\n\nSimilarly to the zero shot case, we have high FactSpotter scores for 1 triples input, and we observe bigger improvements from 2 triples on when adding FactSpotter. \n\nIID FactS Result\n\n\n| Triple Num      | 1     | 2     | 3    | 4    |\n| --------------- | ----- | ----- | ---- | ---- |\n| Number of pairs | 257  | 465  | 63  | 12    |\n| %age of pairs   | 32.65 | 58.34 | 7.90 | 1.50 |\n\nFactSpotter by size of the graph (number of triples):\n|Triple Num|1|2|3|4|\n| --------------- | ----- | ----- | ---- | ---- |\n|T5S|97.75|97.98|98.82|96.96|\n|F-T5S|97.96|98.73|98.83|99.20|\n|T5B|99.55|99.31|99.38|99.77|\n|F-T5B|99.55|99.47|99.38|99.77|\n|JGT-T5|98.09|99.12|98.31|99.77|\n|FGT-T5|98.72|99.35|99.31|99.77|\n\nAs for WebNLG, we observe on the IID split that only the small model benefits from the FactSpotter pluging (T5S compared with F-T5S), in particular on splits of size 2 and larger. \n\n\n**Dart**\n\n|Triple Num|1|2|3|4|5|6|7|\n|-|-|-|-|-|-|-|-|\n|Number of pairs|848|797|821|869|822|598|342|\n|%age of pairs|16.64|15.64|16.11|17.05|16.13|11.73|6.71|\n\nFactSpotter by size of the graph (number of triples):\n\n| Triple Num | 1     | 2     | 3     | 4     | 5     | 6     | 7     |\n| ---------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| T5S| 98.47 | 93.55 | 93.35 | 92.45 | 94.01 | 99.57 | 99.82 |\n| F-T5S| 99.71 | 97.21 | 96.41 | 94.82 | 96.68 | 99.88 | 99.82 |\n| T5B| 98.08 | 95.53 | 94.54 | 94.44 | 96.03 | 99.88 | 99.99 |\n| F-T5B| 99.54 | 96.81 | 96.32 | 96.16 | 97.19 | 99.74 | 99.87 |\n| JGT-T5| 98.49 | 94.36 | 93.61 | 93.44 | 94.33 | 99.82 | 99.80 | \n| FGT-T5| 99.51 | 96.43 | 95.83 | 94.78 | 95.82 | 99.77 | 99.89 |\n\nFor inputs of 1 to 5 triples, all models improve when adding FactSpotter. \nFor inputs of 6 or 7 triples, we observe very high FactSpotter scores. For inputs of size 6, 569 out of 598 triples are from E2E split; for triple num 7, 320 out of 342 are from E2E split. E2E split only has triples about restaurant ratings, while the instances with less triples belong to several different datasets. We will provide in the final paper a more detailed analysis of DART, by dataset and number of triples in the input.  \n\n\n*2.  Relying only on automatic metrics might restrict the results, especially regarding factual consistency which may not be well captured by metrics such as Bleu and METEOR. While the paper executes human inspections on factual consistency, it would be useful to illustrate generated (hallucinated) texts and factual scores exploring distinct dataset.*\n\n\n**Answer:** While BLEU and METEOR cannot measure factfulness, thus cannot reflect the effectiveness of our work, we report them for consistency with prior works (\"JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphs\", and \"Investigating Pretrained Language Models for Graph-to-Text Generation\"). We also report BLEURT, BART and Bert-F1 as SOTA metrics. \nWe show **examples of hallucinations and how our method improves G2T models** in the Annexes, Table 8. The types of hallucinations that we have observed and that our work can address are the following:\n - wrong entities in subject or object\n - missing entities in subject or object\n - wrong predicate\n - missing predicate\n - missing triple\n\n This is possible through the construction of the negative set described in Section 4. We plan to devote more space to illustrating such hallucinations, in our final version. \n\n### Missing references\nWe thank the reviewer for pointing out the missing reference. \nIn “Modelling Global and Local Node Contexts for Text Generation from Knowledge Graphs”, the authors designed a network which encodes both global and local information for text generation. JointGT, one of the baselines we used, can also represent both global and local information for data-to-text generation. The global information is from the original attention module of Transformers. Apart from that, the Structure-Aware Self-Attention Layer in JointGT can also capture local information.  Although various baselines designed neural networks to encode both global and local information, this does not guarantee that all facts in the input knowledge graph are correctly verbalised in the generated text. Hence, we think it’s useful to use FactSpotter to choose a better beam when decoding. \nWe will add this discussion in our paper.  \n\n### Typos Grammar Style And Presentation Improvements\n\nWe thank the reviewer's suggestions to the writing of our paper. We'll improve it according to the suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This work focuses on measuring and improving factual consistency of graph-to-text models, which is an important factor when realizing structure data (e.g., triples into natural language).", "reasons_to_reject": "- It would be good to see ablation studies and qualitative analysis for examples that the proposed metric improves during the decoding. For example, in examples that contains more facts (e.g., 5 triples) the model might have more difficulty generating factually generated text. Investigating such cases could bring insights and better illustrate the metric capabilities. \n- Relying only on automatic metrics might restrict the results, especially regarding factual consistency which may not be well captured by metrics such as Bleu and METEOR. While the paper executes human inspections on factual consistency, it would be useful to illustrate generated (hallucinated) texts and factual scores exploring distinct dataset.", "questions_to_authors": "- I understand that the metric receives as input a triple and a text that contains the fact described in the triple but also describes other facts. Why don't the authors consider the whole graph as input instead of using a unique triple? Having a unique input for the graph would benefit the decoding?\n- Do you have some thoughts on why the T5 base models have worse performance on FactS compared to small models (Table 4)? I would expect that larger models would generate more faithful texts to the input graph.", "paper_topic_and_main_contributions": "This paper presents a metric that aims to capture the faithfulness of a generated text compared to the fact present in a given triple. This metric is realized using a classifier based on a pretrained LM and trained with pairs or triples and texts with both positive and negative instances (that are created synthetically from the corpus perturbing the original triple or corresponding text).\n\nThe method is evaluated on different QA and Data-to-text datasets. It achieves strong correlation with human annotations for factors such as correctness, coverage and relevance. The paper also explores the use of the metric during decoding time to guide the generation to be more faithful to the input.", "excitement": 2, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "- Modeling Global and Local Node Contexts for Text Generation from Knowledge Graphs (Ribeiro et al., TACL 2020)", "Typos_Grammar_Style_and_Presentation_Improvements": "- Factual faithfulness does not seem a little verbose to describe the factual quality of the generated text. Maybe factual consistency or simple faithfulness would be some suggestions."}, "extra_text": {}}, {"review_id": "I4dulOBFDC", "reviewer": "Reviewer_wiwF", "summary": "", "questions": "Could this be easily extended to data-to-text tasks that use different data representations to RDF?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback and suggestions! Below, we reiterate the reviewer's point and present our answer.\n\n*Could this be easily extended to data-to-text tasks that use different data representations to RDF?*\n\n**Answer:** Our work can be easily extended to data-to-text tasks. The DART dataset on which we run experiments contains a table-to-text split in which tables were represented as sets of triples. There are known methods for transforming a relational dataset into RDF, e.g., the R2RML language (in paper \"R2RML: RDB to RDF Mapping Language\"). We can train a FactSpotter with a self-supervised approach for any data-to-text task, and use it to encourage factual generations. Our FactSpotter also has capability under zero-shot settings, as long as the format is similar to training data. We will comment on this in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a well presented paper, with the authors very clearly explaining their motivations and the steps they take to develop and test their metric. The steps taken are intuitive and appear to yield strong results in evaluating factuality for this task. Their proposal for a method of integrating the metric into the model during decoding to promote generations that better contribute to factuality wrt the triple that needs to be lexicalized is creative and seems to offer good improvements to performance with little downside (e.g. no need to retrain the generative model). ", "reasons_to_reject": "The fact that this is built specifically around RDF triples as fact representations means that its applicability is likely limited to this specific task. It is also limited to an exhaustive set of predicates that happen to exist in the training data.", "questions_to_authors": "Could this be easily extended to data-to-text tasks that use different data representations to RDF?", "paper_topic_and_main_contributions": "The authors investigate factual faithfulness within graph-to-text generation and propose a new metric, FactSpotter, which determines whether a given RDF triple is present in a generated text and can also be integrated into a generative model to improve factuality in outputs. They also consider the remaining difficulty of the task for state-of-the-art systems.\n\nTheir metric, FactSpotter, is basically trained to perform binary classification of whether a triple is mentioned properly within a text. The real-valued score is the result of the final softmax layer in the network, averaged over the set of facts for some graph-text pair. Ultimately, this appears to correlate well with human judgements of semantic adequacy, data coverage, etc. and better than most existing metrics. \n\nThey encourage more factual generations by modifying the prediction step such that new tokens are more likely to be added to the output if they increase the likelihood of better representing some fact. I.e. tokens will be reweighted according to their contribution to factuality wrt some fact until it is probable that the fact is already represented in preceding output text. On most datasets, integrating FactSpotter into modelling in this way allows for better factuality scores without much degradation (and sometimes further improvement) of other metrics.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "NA"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new metric, FactSpotter, to evaluate factual faithfulness by how the generated text covers facts in the knowledge base (KB). FactSpotter uses a classifier based on a pretrained LM and trained with pairs or triples and texts with both positive and negative examples. By using ELECTRA, FactSpotter outperformed conventional metrics in correlation to human evaluation results on the 2017 and 2020 WebNLG graph-to-text (G2T) challenges. Due to the necessity of the knowledge base, there is a consideration that FactSpotter may be restricted to specific tasks. However, the authors' response to Reviewer wiwF's question refers to the extension of FactSpotter by converting tables into knowledge bases. Since this approach partially solves the problem, we can judge that the paper should be accepted as a findings paper.
Decision: Accept-Findings