# FactSpotter: Evaluating the Factual Faithfulness of Graph-to-Text Generation

Kun Zhang, Oana Balalau, Ioana Manolescu

<figure><img src="image_1.png" /></figure>

## To cite this version:

Kun Zhang, Oana Balalau, Ioana Manolescu. FactSpotter: Evaluating the Factual Faithfulness of Graph-to-Text Generation. Findings of EMNLP 2023 - Conference on Empirical Methods in Natural Language Processing, Dec 2023, Singapore, Singapore. [https://hal.science/hal-04257838v1](https://hal.science/hal-04257838v1) [https://hal.science/hal-04257838v1](https://hal.science/hal-04257838v1)

Submitted on 25 Oct 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# FactSpotter: Evaluating the Factual Faithfulness of Graph-to-Text Generation

Kun Zhang<fnref n="1" />,² and Oana Balalau<fnref n="1" />,² and Ioana Manolescu<fnref n="1" />,²

¹Inria, ²Institut Polytechnique de Paris

## Abstract

Graph-to-text (G2T) generation takes a graph as input and aims to generate a fluent and faithful textual representation of the information in the graph. The task has many applications, such as dialogue generation and question answering. In this work, we investigate to what extent the G2T generation problem is solved for previously studied datasets, and how proposed metrics perform when comparing generated texts. To help address their limitations, we propose a new metric that correctly identifies factual faithfulness, i.e., given a triple (subject, predicate, object), it decides if the triple is present in a generated text. We show that our metric *FactSpotter* achieves the highest correlation with human annotations on data correctness, data coverage, and relevance. In addition, *FactSpotter* can be used as a plug-in feature to improve the factual faithfulness of existing models. Finally, we investigate if existing G2T datasets are still challenging for state-of-the-art models. Our code is available online: https://github.com/guihuzhang/FactSpotter.

## 1 Introduction

Graph-to-text (G2T) generation is an important task in natural language generation, as it renders graphs, and in particular knowledge graphs, accessible to non-technical users in downstream applications such as question answering (Gu et al., 2021), (Romero and Razniewski, 2020), knowledge-grounded dialogue generation (Zhou et al., 2018), and document summarization (Fan et al., 2019). In recent years, there have been several datasets (Gardent et al., 2017; Nan et al., 2021) and methods proposed for G2T generation (Ke et al., 2021; Ribeiro et al., 2021), in addition to G2T competitions (Shimorina et al., 2018; Castro Ferreira et al., 2020). Evaluating text generation is a challenging task in itself (Celikyilmaz et al., 2020); moreover, in the context of G2T generation, we are concerned not only with the fluency of the gener-

ated text, but also with its faithfulness to the input graph. While recent models such as T5 and GPT models are very fluent, they have been criticized for their factual accuracy, a problem commonly referred to as hallucination (Ji et al., 2023; Liu et al., 2022). Hallucinations are a serious drawback in G2T, where the generated text should only contain facts mentioned in the input graph.

In this work, we focus on measuring and improving the factual accuracy of G2T generative models. More precisely, our contributions are as follows: i) We introduce a novel metric *FactSpotter* for detecting if G2T generations are faithful to the facts present in the input graph; ii) We show how *FactSpotter* can be used in the inference step of any G2T model to improve its generations; iii) We analyze the difficulty of existing G2T datasets and determine which are (resp., are no longer) challenging for state-of-the-art models. *FactSpotter* can be extended to other data-to-text tasks via methods for transforming a relational dataset into RDF, such as the R2RML language<fnref n="1" />.

## 2 Related Work

### 2.1 Graph-to-text (G2T) generation

In (Ribeiro et al., 2021), the authors investigate the potential of pretrained language models (PLM) on the G2T task. They consider two Transformer-based models (Vaswani et al., 2017) with an encoder-decoder structure: T5 (Raffel et al., 2020), and Bart (Lewis et al., 2020). The models receive in input a linearized (serialized) version of the input graph, in which they add the tags $\langle H \rangle$, $\langle R \rangle$, and $\langle T \rangle$ before the head entity, the relation, and tail entity of a triple, respectively.

The potential of large language models is further investigated in (Keymanesh et al., 2022) on the DART dataset (Nan et al., 2021), a data-to-text dataset constructed from tables, available in

<footnote n="1">¹http://www.w3.org/TR/r2rml/</footnote>
a triple-to-sentence format. The dataset is con-
structed from a combination of manual and auto-
matic techniques. The authors empirically evaluate
the GPT2 model (Radford et al., 2019) and the T5
model on the dataset by varying the amount of su-
pervision a model receives. They also investigate
the potential of adding predicate descriptors in the
prompt and re-ranking generations. In a small-scale
human evaluation, they find that their best model,
T5 large, outperforms the reference text regarding
hallucinations and fluency, underlining that *exist-
ing datasets suffer from poor human annotations*,
which we also observe and discuss in Section 7.

The authors of (Ke et al., 2021) propose modify-
ing the Transformer model by adding extra atten-
tion and pooling layers to improve G2T generation.
In addition, the model’s pretraining has three steps
given the input graph and the expected output text:
1) reconstructing the text sequence given the com-
plete subgraph, 2) predicting the masked entities
and relations in the corrupted subgraph, given the
complete text, and, 3) aligning the embedding vec-
tors of the knowledge graph and the text.

Most state-of-the-art G2T models are based on
Transformers, and they can generally generate flu-
ent texts related to given graphs. Although various
baselines designed neural networks to encode both
global and local information (Ribeiro et al., 2020),
they cannot guarantee that generated texts are fac-
tual faithful to the given graphs. It's also not clear
whether current G2T datasets are still challenging.

## 2.2 Evaluation metrics for text generation

Alongside the significant improvements that mod-
els for G2T generation underwent and, in general,
the improvement of language models, new metrics
to assess the generations’ quality have been pro-
posed (Celikyilmaz et al., 2020; Sai et al., 2022).
G2T generation belongs to the broader field of natu-
ral language generation, including tasks such as ma-
chine translation, automatic summarization, ques-
tion answering, and more. Each task has specific re-
quirements, which might entail using some metrics
over others. For example, in machine translation,
the translation should match the ground-truth text
as closely as possible, while in chatting or summa-
rization, adding or removing some information is
acceptable or even desirable.

Evaluation metrics are split into three categories: *human-centric metrics*, *untrained automatic metrics*, and *machine-learned metrics*. Human evalu-

ation is the most important of these metrics. It
consists of asking users to evaluate the quality
of a text in a specific category, such as fluency
or correctness. Unfortunately, human evaluation
sometimes suffers from a low inter-annotator agree-
ment (Celikyilmaz et al., 2020), (Belz and Reiter,
2006) as different people might have different no-
tions of what makes a text fluent or correct, or
the instructions they receive for annotation might
lack sufficient clarity. Also, human annotation
is time and money-consuming; it can represent a
bottleneck in the iterative process of improving a
model. Hence the need for automatic metrics such
as BLEU (Papineni et al., 2002), ROUGE (Lin,
2004), METEOR (Banerjee and Lavie, 2005),
BERTScore (Zhang et al., 2019), BLEURT (Sellam
et al., 2020) BARTScore (Yuan et al., 2021), In-
foLM (Colombo et al., 2022), among others. Over-
lap measures based on *n*-grams, such as BLEU,
ROUGE, and METEOR, have been widely used
in the literature, while recently proposed metrics
based on *word embeddings*, such as BERTScore,
BLEURT, BARTScore are gaining traction. The
embedding-based measures have been shown to
correlate better with human evaluation than the *n*-
grams metrics. In addition, some metrics, such as
BLEURT, have been trained on human annotations.
As these automatic metrics are based on distances
or similarities to ground-truth texts, they rely on
the quality of annotated sentences.

Apart from the works mentioned above, a few
prior studies assessed factual faithfulness in graph-
to-text generation. In (Faille et al., 2021) the au-
thors introduce a metric for verifying whether en-
tities in input graphs are represented in the gener-
ated texts. However, this work does not evaluate
the quality of the predicates in the generated texts,
which is a much more difficult task. In (Rebuffel
et al., 2021), a question generation (QG) and ques-
tion answering (QA) framework is employed to
evaluate the quality of the generated text by deter-
mining if each question posed by the QG module
can be addressed by the QA module. We believe
our contribution can further advance the state-of-
the-art as: i) *FactSpotter* requires significantly less
computational resources; ii) *FactSpotter* is self su-
pervised, thus it does not requires additional data
to the G2T model; item *FactSpotter* can be plugged
into a G2T to improve its generation.
# 3 Problem Statement

A knowledge graph (KG) consists of facts (or data triples) of the form $\langle subject, relation, object/literal \rangle$ and/or an ontology describing the properties that hold between classes and/or relations.

**Graph-to-text.** A G2T tool takes in input a graph and outputs a textual representation of the graph. G2T inputs are often subgraphs of larger real-world graphs, such as knowledge graphs. The textual representation should be fluent and should contain all the facts present in the input graph. For example, given a subgraph consisting of the single DBpedia (Auer et al., 2007) fact $\langle \text{The Myth of Sisyphus, author, Albert Camus} \rangle$, we would like to generate the sentence *The Myth of Sisyphus was written by Albert Camus*. This work primarily focuses on creating textual representations of KG subgraphs.

**Factual faithfulness.** The following human criteria have been proposed to evaluate the factual quality of a generated text (Castro Ferreira et al., 2020): data correctness, data coverage, and relevance. Given a graph *G* and a machine-generated text *T*, the generated text is characterized by:

1. *Data coverage or recall:* are all the descriptions presented in the graph *G* included in the text *T*? i) *Covered predicates:* does *T* contain all the predicates from *G*? ii) *Covered entities:* does *T* contain all the entities from *G*?

2. *Relevance or precision:* i) *Relevant predicates:* does *T* only contain relevant predicates? ii) *Relevant entities:* does *T* only contain relevant entities?

3. Correctness: are predicates correctly mentioned and adequately introduced in the data?

**Research questions.** We focus on the following three research questions:

**RQ1** What metric would better correlate with the factual faithfulness of generated text?

**RQ2** Can we improve factual faithfulness of G2T?

**RQ3** Is the G2T task solved on existing datasets?

# 4 FactSpotter: An explainable metric for factual faithfulness

In this section, we introduce a new metric for factual faithfulness. A good metric should be **interpretable**, that is: given in input a fact *f* of the form

$\langle \text{subject, predicate, object} \rangle$ and a text *T*, it assigns a score between 0 and 1, where a score close to 0 signifies that the text that does not correctly represent the fact, and a score close to 1 rewards a factual faithful textual representation of *f* in *T*. Such a metric can be used to compare different generation systems and, in addition, assess a single system on a given dataset, to determine how close the system is to representing the input graphs correctly.

The **intuition of our score** is the following. We train a model to perform a task simpler than G2T generation: only *detecting whether facts exist in a sentence* and *whether they are well expressed*. This simpler model can then be used as a **plug-in feature** by any existing G2T model, to aid it perform the more complex task of language generation.

Our metric, *FactSpotter*, is trained as a binary classifier. Given in input a fact $\langle \text{subject, predicate, object} \rangle$ and a sentence, it should predict 1 if the fact is well expressed in the sentence, or 0 otherwise. Thus, *FactSpotter* is inherently interpretable. We leverage as classification models recent large language models, capable of detecting semantic closeness, even if different words, e.g., synonyms, are used. This approach is similar to the one taken to compute metrics such as *BertScore* and *BartScore*. Given an input G2T dataset *D*, with a *training* (train) set, a *development* (dev) set, and a *test* set, we create the training set as follows:

**Positive samples.** Given an instance of the training set of the form (graph *G*, ground truth text *T*), for each fact (triple) in *G*, we create a positive sample of the form (fact *f*, ground truth text *T*).

**Negative samples.** Given an instance of the training set of the form (graph *G*, ground truth text *T*), for each fact *f* ∈ *G*, we create negative samples as follows: i) Type I: we perturb the fact *f*: we change its predicate, or an entity (subject or object), or both, while the ground truth text *T* remains unchanged. ii) Type II: we perturb the ground truth text *T*: we drop one or both entities related to *f* from the text, or drop the *n*-grams most similar to the predicate of *f*, or we apply simultaneously several modifications, keeping the fact unchanged.

For example, given the fact $\langle \text{The Myth of Sisyphus, author, Albert Camus} \rangle$ and its associated text "The Myth of Sisyphus was written by Albert Camus", a Type I negative sample alters the fact ($\langle \text{The Myth of Sisyphus, author, Simone de Beauvoir} \rangle$), "The Myth of Sisyphus was written by Albert Ca-
mus”), while for Type II yields the sample (〈The Myth of Sisyphus, author, Albert Camus〉, “The Myth of Sisyphus was written”). We associate probabilities to each perturbation, and control the generation such that for each positive sample, we only generate one negative sample. To allow our classifier to learn from different negative samples and avoid over fitting (Chen et al., 2020), for each training epoch, we use a newly generated set of negative samples. The development set is built in the same way. Through evaluation on a fixed test set (Appendix A.4.1) we find that the model which best detects factual faithfulness is the one with the highest probability of perturbing the fact (90%) and 10% the probability of perturbing the ground truth.

Above, we have described *FactSpotter* as a (*trained*) classifier. To use it as a score (*metric*), we take the output of the model after the softmax layer. The final score of a generated text *T* given an input graph *G* is the average over the scores for each pair (fact *f* ∈ *G*, generated text *T*).

**Parameters and performance of FactSpotter.**

As we aim to add our metric, *FactSpotter*, in the inference step of graph-to-text generation, we prefer small language models. Hence, we select the small Electra model (Clark et al., 2020). We have experimented with other small models, such as DistilBERT and DistilRoBERTa, but we did not observe an improvement. We train our classifier for 16 epochs, with a learning rate of 5 · 10⁻⁵, and the AdamW optimizer. We describe in the Appendix A.4.1 how we chose the percentages of negative samples for *FactSpotter*.

The performance of *FactSpotter* on the test splits across multiple datasets is detailed in Table 1, with accuracy and F1 score. We also report numbers of true positives/negatives (TP/TN), and false positives/negatives (FP/FN) in the table.

<table><caption>Table 1: Performance of *FactSpotter* on test splits.</caption><thead><tr><th>Dataset</th><th>Acc.</th><th>F1</th><th>TP/TN</th><th>FP/FN</th></tr></thead><tbody><tr><td>GrailQA</td><td>96.85</td><td>96.85</td><td>2272/2643</td><td>64/96</td></tr><tr><td>SimpleQ.</td><td>95.41</td><td>95.41</td><td>10260/10438</td><td>419/575</td></tr><tr><td>DART</td><td>97.62</td><td>97.62</td><td>26613/26352</td><td>607/684</td></tr><tr><td>WebNLG17</td><td>99.10</td><td>99.09</td><td>8594/8575</td><td>93/63</td></tr><tr><td>WebNLG20</td><td>95.21</td><td>95.21</td><td>9637/9853</td><td>317/663</td></tr></tbody></table>

**5 Evaluating graph-to-text generation**

We investigate our first research question (RQ1): which metrics would better correlate with the fac-

tual faithfulness of the generated text? For this question, we compare *FactSpotter* with: BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), BERTScore (Zhang et al., 2019), BLEURT (Sellam et al., 2020), BARTScore (Yuan et al., 2021). The only metric that is not normalized is BARTScore. This metric has a variant specifically adapted for factual faithfulness, BARTScore-faithful. Further details of these metrics are provided in Appendix A.1.

We calculate the *system-level correlation* between automatic metrics and human annotations. Given *S* systems under evaluation, for a certain dimension, e.g., fluency, we compute the correlation between the system-level automatic metric scores [*M*(*S*₁),..., *M*(*S*ₙ)] and the corresponding system-level human scores [*H*(*S*₁),..., *H*(*S*ₙ)], where *M*(*S*ᵢ) is the score of the automatic metric on the texts from system *S*ᵢ, and *H*(*S*ᵢ) is the score of the human annotation on the same result. Similarly to (Colombo et al., 2022), we compute three correlation metrics: Pearson correlation (*r*), Spearman correlation (*ρ*), and Kendall's Tau (*τ*). To test if a metric *M*₁ has a higher correlation with human annotations than *M*₂, we use the bootstrapping technique proposed in (Wilcox, 2016), which we describe in the Appendix A.2. We also report the sentence-level correlation in Appendix A.5. In addition to automatic measures, we report the correlation between one annotator and the average score of the remaining annotators, which should be a upper bound on the correlation we can obtain using automatic measures.

**WebNLG 2017.** In the WebNLG 2017 challenge (Shimorina et al., 2018), the organizers annotated 9 submissions on *semantic adequacy* (the text correctly represents the meaning in the data), *text structure* (as above, referred in the original paper as grammar) and *fluency* (as above). This annotation has carried over 223 samples.

**WebNLG 2020.** After the WebNLG 2020 Challenge (Castro Ferreira et al., 2020), the organizers annotated the 16 participating systems on *data correctness* (the predicates found in the data are correctly mentioned together with their subject and object), *data coverage* (the text includes descriptions of all predicates presented in the data), and *relevance* (the text describes only those predicates with related subjects and objects which are in the data), in addition to *text structure* (the text is grammati-
<table><caption>Table 2: Correlation at the system level with human judgment on correctness, data coverage, relevance, fluency and text structure for the 2020 WebNLG task. For tables here and below, BERTF1 stands for BERTScore-F1, BARTS for BARTScore, BARTS-F for BARTScore-faithful, FactS for FactSpotter. We **highlight** the best result and we mark it with an asterisk when it is statistically significantly larger than any other metric (excluding Human to Human correlation). FactSpotter performs the best on correctness, data coverage, and relevance. We put a value for correlation if the p-value *p* < 0.05.</caption><thead><tr><th rowspan="2">Metric</th><th colspan="3">Correct.</th><th colspan="3">D. Cover.</th><th colspan="3">Relev.</th><th colspan="3">Fluency</th><th colspan="3">T. Struct.</th></tr><tr><th>r</th><th>ρ</th><th>τ</th><th>r</th><th>ρ</th><th>τ</th><th>r</th><th>ρ</th><th>τ</th><th>r</th><th>ρ</th><th>τ</th><th>r</th><th>ρ</th><th>τ</th></tr></thead><tbody><tr><td>Correct.</td><td>1.0</td><td>1.0</td><td>1.0</td><td>0.96</td><td>0.81</td><td>0.66</td><td>0.97</td><td>0.81</td><td>0.66</td><td>0.80</td><td>0.77</td><td>0.60</td><td>0.79</td><td>0.76</td><td>0.59</td></tr><tr><td>D. Cover.</td><td></td><td></td><td></td><td>1.0</td><td>1.0</td><td>1.0</td><td>0.93</td><td>0.80</td><td>0.64</td><td>0.71</td><td>0.56</td><td>0.43</td><td>0.69</td><td>0.55</td><td>0.42</td></tr><tr><td>Relev.</td><td></td><td></td><td></td><td></td><td></td><td></td><td>1.0</td><td>1.0</td><td>1.0</td><td>0.76</td><td>0.63</td><td>0.48</td><td>0.76</td><td>0.62</td><td>0.47</td></tr><tr><td>Fluency</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>1.0</td><td>1.0</td><td>1.0</td><td>0.98</td><td>0.97</td><td>0.91</td></tr><tr><td>T. Struct.</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>1.0</td><td>1.0</td><td>1.0</td></tr><tr><td>Human</td><td>0.96</td><td>0.80</td><td>0.65</td><td>0.93</td><td>0.83</td><td>0.68</td><td>0.96</td><td>0.74</td><td>0.59</td><td>0.95</td><td>0.93</td><td>0.80</td><td>0.93</td><td>0.91</td><td>0.77</td></tr><tr><td>BLEU</td><td>0.59</td><td>0.64</td><td>0.48</td><td>0.53</td><td>0.53</td><td>0.40</td><td>0.56</td><td>0.60</td><td>0.45</td><td>0.87</td><td>0.84</td><td>0.68</td><td>0.86</td><td>0.84</td><td>0.68</td></tr><tr><td>METEOR</td><td>0.72</td><td>0.75</td><td>0.60</td><td>0.65</td><td>0.58</td><td>0.44</td><td>0.70</td><td>0.64</td><td>0.50</td><td>0.88</td><td>0.89</td><td>0.74</td><td>0.86</td><td>0.88</td><td>0.72</td></tr><tr><td>BERTF1</td><td>0.83</td><td>0.77</td><td>0.60</td><td>0.74</td><td>0.58</td><td>0.43</td><td>0.81</td><td>0.65</td><td>0.50</td><td><b>0.90</b></td><td><b>0.93</b></td><td><b>0.80</b></td><td>0.88</td><td><b>0.92</b></td><td><b>0.78</b></td></tr><tr><td>BLEURT</td><td>0.93</td><td>0.82</td><td><b>0.67</b></td><td>0.86</td><td>0.65</td><td>0.50</td><td>0.91</td><td>0.69</td><td>0.55</td><td><b>0.90</b></td><td>0.92</td><td>0.78</td><td><b>0.89</b></td><td>0.91</td><td>0.76</td></tr><tr><td>BARTS</td><td>0.90</td><td><b>0.83</b></td><td><b>0.67</b></td><td>0.86</td><td>0.71</td><td>0.53</td><td>0.88</td><td>0.71</td><td>0.56</td><td>0.77</td><td>0.81</td><td>0.63</td><td>0.75</td><td>0.80</td><td>0.62</td></tr><tr><td>BARTS-F</td><td>0.67</td><td>0.54</td><td>0.41</td><td>0.68</td><td>0.61</td><td>0.46</td><td>0.68</td><td>0.59</td><td>0.45</td><td>0.51</td><td>-</td><td>-</td><td>0.52</td><td>-</td><td>-</td></tr><tr><td>FactS</td><td><b>0.94</b></td><td>0.80</td><td>0.64</td><td><b>0.91</b></td><td><b>0.87</b></td><td><b>0.71</b></td><td><b>0.96</b><sup>*</sup></td><td><b>0.79</b></td><td><b>0.64</b></td><td>0.74</td><td>0.59</td><td>0.45</td><td>0.72</td><td>0.59</td><td>0.45</td></tr></tbody></table>

<table><caption>Table 3: Correlation at the system level with human judgment on semantic adequacy, grammar, and fluency, for the 2017 WebNLG dataset.</caption><thead><tr><th rowspan="2">Metric</th><th colspan="3">Sem. Adeq.</th><th colspan="3">T. Struct.</th><th colspan="3">Fluency</th></tr><tr><th>r</th><th>ρ</th><th>τ</th><th>r</th><th>ρ</th><th>τ</th><th>r</th><th>ρ</th><th>τ</th></tr></thead><tbody><tr><td>Sem. Adeq.</td><td>1.0</td><td>1.0</td><td>1.0</td><td>0.73</td><td>0.65</td><td>0.52</td><td>0.71</td><td>0.66</td><td>0.52</td></tr><tr><td>T. Struct.</td><td></td><td></td><td></td><td>1.0</td><td>1.0</td><td>1.0</td><td>0.98</td><td>0.95</td><td>0.88</td></tr><tr><td>Fluency</td><td></td><td></td><td></td><td></td><td></td><td></td><td>1.0</td><td>1.0</td><td>1.0</td></tr><tr><td>Human</td><td>0.99</td><td>0.98</td><td>0.94</td><td>0.98</td><td>0.92</td><td>0.84</td><td>0.98</td><td>0.89</td><td>0.79</td></tr><tr><td>BLEU</td><td>0.76</td><td>0.71</td><td>0.56</td><td><b>0.86</b></td><td><b>0.71</b></td><td><b>0.57</b></td><td><b>0.83</b></td><td><b>0.72</b></td><td><b>0.57</b></td></tr><tr><td>METEOR</td><td>0.86</td><td>0.83</td><td>0.67</td><td>0.85</td><td>0.70</td><td><b>0.57</b></td><td>0.80</td><td>0.70</td><td>0.56</td></tr><tr><td>BERTF1</td><td>0.70</td><td>0.78</td><td>0.63</td><td>0.71</td><td>0.70</td><td><b>0.57</b></td><td>0.69</td><td>0.70</td><td><b>0.57</b></td></tr><tr><td>BLEURT</td><td>0.90</td><td>0.88</td><td>0.72</td><td>0.84</td><td>0.69</td><td>0.56</td><td>0.79</td><td>0.68</td><td>0.56</td></tr><tr><td>BARTS</td><td>0.90</td><td>0.87</td><td>0.78</td><td>0.71</td><td>-</td><td>-</td><td>0.68</td><td>-</td><td>-</td></tr><tr><td>FactS</td><td><b>0.97</b><sup>*</sup></td><td><b>0.93</b></td><td><b>0.85</b></td><td>0.67</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr></tbody></table>

cal, well structured, and written in good English) and *fluency* (the text progresses naturally, forms a coherent whole and is easy to understand). 178 generations of each system were annotated.

**Results.** On WebNLG 2020, Table 2 shows that FactSpotter has the best performance on factual faithfulness, significantly improving *relevance*. BLEU, METEOR, BERTScore and BLEURT reach similar *fluency* and *text structure* scores. For the results on WebNLG 2017 in Table 3, FactSpotter has the highest performance on *semantic adequacy*, which is the only dimension related to factual faithfulness. For *text structure* and *fluency*, BLEURT obtains the best results, although the results are not statistically significant. Overall, previous metrics are better on *text structure* and *fluency*, which are generally considered as no longer a bottleneck for large language models. FactSpotter is the best suited metric on factual faithfulness.

# 6 Improving the factual faithfulness of graph-to-text generation

In this section, we investigate the answer to our third research question: can we improve graph-to-text generation on factual faithfulness (RQ2)? For this, we first explain how to improve the inference step of any G2T model using FactSpotter, and then we present the results of this technique on the state-of-the-art models for G2T generation.

## 6.1 Improving models’ factual faithfulness

Let $\mathcal{M}$ be a neural network G2T (seq2seq) model that, given an input sequence $x = x_1, x_2, \dots, x_M$, produces an output sequence $y = y_1, y_2, \dots, y_N$, where $x_i \in V_x$, $y_i \in V_y$, and $V_x, V_y$ are the vocabularies from where we select items in the input, respectively output sequence. In the inference step, the model generates the sequence $y$ that maximizes:

$$P(y|x) = \prod_{i=1}^{N} P(y_i|y_{<i}, x)$$

In practice, for computational efficiency, the log of the probabilities are typically utilized in beam search. We use the following method to improve factual faithfulness in G2T inference *without re-training the model*.

Given: i) a graph-to-text generation model $\mathcal{M}$, ii) our factual faithfulness classifier, i.e., FactSpotter, iii) a subgraph $G$ composed of $F$ facts, we encourage factual generations by modifying the
prediction step as follows:

$$ \log(P^f(y_i|y_{<i}, x)) = \lambda \sum_{j=1}^{F} (1 - P_{\text{fact}_j}(y_{<i-1})) \log P_{\text{fact}_j}(y_{<i}) + \log(P(y_i|y_{<i}, x)) \quad (1) $$

where: i) $P^f(y_i|y_{<i}, x)$ is the probability of generating token $y_i$ given the factual classifier; ii) $P_{\text{fact}_j}(y_{<i-1})$ is the probability of correctly representing the fact j in the previously generated tokens $y_0, ..., y_{i-1}$, computed by FactSpotter. iii) $P(y_i|y_{<i}, x)$ is the probability for generating the next token based on previous $i-1$ tokens.

In our equation, a fact j is encouraged only if we have not observed in the text generated until step i, according to $P_{\text{fact}_j}(y_{<i-1})$. Then adding token $y_i$ would increase the probability of the text including the fact j, $P_{\text{fact}_j}(y_{<i})$. When $P_{\text{fact}_j}(y_{<i-1})$ is small, the equation encourages the selection of words belonging to the fact j. As P is large and $1 - P_{\text{fact}_j}(y_{<i-1})$ tends to 0, then we can consider that the fact j already appears in the text, and words that satisfy fact j will no longer be encouraged. The weight $\lambda$ controls the influence of the FactSpotter on the prediction of the following words. A high $\lambda$ might yield a factual text, but not necessarily a fluent one. We generate tokens till we have generated text $S = y_0, ..., y_k$ for which $\forall j \in F, P_{\text{fact}_j}(S) > 0.5$, i.e., the probability of each fact j in G is verbalized in the text S is over 0.5, the standard positive threshold.

## 6.2 Models

We consider for our evaluation state-of-the-art models for G2T generation, PLM (Ribeiro et al., 2021) and JointGT (Ke et al., 2021). The former investigates how a standard seq2seq model can perform on G2T, given a carefully constructed representation of a graph. The latter proposes a more complex neural network, with built-in attention layers specialized on graph structured inputs. Both are initialized with the pretrained weights of a language model. Similar to the authors, we consider in this work T5 (Raffel et al., 2020) for both G2T models. For simplicity we refer to the first model as T5, and to the second model as JointGT. We refer to the models modified as explained in Section 6.1 as FactT5 and FactJointGT. For T5 and FactT5, we initialize the weights with T5 small, and T5 base. For JointGT and FactJointGT we initialize

the weights with a pretrained T5 base model offered by the authors of JointGT. For the fine-tuning step (each model is fine-tuned on the training split of the datasets), we train the small models with a learning rate of $10^{-4}$ and a batch size of 32, while the base models are trained with a learning rate of $5 \cdot 10^{-5}$ and a batch of 16. We use a beam size of 5 and the AdamW optimizer for both sizes of models. For Equation 1, we fix the weight $\lambda = 0.15$ (parameter tuning in Appendix A.4.2).

## 6.3 Datasets

To evaluate G2T performance, we need (graph, text) pairs datasets. The graphs can be directly extracted from an underlying knowledge graph or adapted to have richer semantics, such as query graphs (Yih et al., 2015). The text associated to the graph should be a sentence or a paragraph verbalizing all the information contained in the subgraph.

Several datasets are proposed in the literature for the G2T task, such as WebNLG (Gardent et al., 2017). Many question-answering (QA, in short) datasets are also in the form (graph, text). Since question answering datasets can be very large and cover many KG predicates (Gu et al., 2021), we also leverage such datasets. To ensure that FactSpotter has never encountered the test data, it is trained exclusively on the training set and evaluated on the validation split.

**SimpleQuestions** (Bordes et al., 2015) is a QA dataset built on Freebase (Bollacker et al., 2008). The dataset contains 108K (triple, question) pairs, where the question corresponds to the subject and predicate from the triple, and the answer is the object of the triple. For example, given the triple (Gulliver's Travels, book/written_work/author, Dean Swift), the question is Who wrote Gulliver's Travels?, with the answer Dean Swift. The dataset covers 76 domains, 741 classes, 89K entities and 2K relations. A Freebase domain is a general area of knowledge such as business, politics, economics, etc. We created our own split for this dataset, where the test set is zero shot: we have not seen the predicates during training. FactSpotter can be trained to correctly classify if a question refers to a triple, even if the object or subject is missing from the question, as we replace the entity with its type.

**GrailQA** (Gu et al., 2021) is also a QA dataset that uses Freebase, created using human annotation. The original dataset contains 64K (triple, question) pairs, however, the test set is not released
as the authors have created a QA challenge<fnref n="2" />, hence we use the development set as a test set. The remaining data (training and development) consists of 51K pairs. The authors propose three levels of generalization and split the development and test as follows. 50% of the pairs from held-out domains (Freebase assigns to each entity and predicate a domain, such as music, sports, etc.) are not covered in training: this is the zero-shot split. 25% of the pairs correspond to graphs where the combination of ontology items (classes and predicates) were not covered in training: this is the compositional split. Finally, the remaining 25% are randomly sampled from the same distribution as the training dataset: the i.i.d. split. The i.i.d. and compositional subsets contain only ontology items (classes and predicates) covered in training. For the zero-shot subset, five domains are held out for validation.

**WebNLG** (Gardent et al., 2017; Castro Ferreira et al., 2020) is a text generation dataset on DBPedia, created via human annotation. The dataset consists of (graph, paragraph) pairs, where the graph is a set of DBPedia facts, and the paragraph consists of one or several sentences that describe the graph. We use the 2017 (2.1) and 2020 (3.0) versions of the dataset<fnref n="3" />. The 2017 version of the dataset contains 42K graph-text pairs, and it has two splits, the standard version and the constrained version. In the constrained version, the test set does not contain that a triple occurring in train/dev. In this work we considered only the constrained split, as it is more challenging. The WebNLG 2020 dataset has 40K pairs, which comprises 10 categories that were previously seen and utilized in WebNLG 2017, as well as 5 categories that were unseen in WebNLG 2017 are now incorporated into the seen data of the WebNLG 2020 dataset. It also introduces a new category of data: company.

**DART** (Nan et al., 2021) is a data-to-text dataset based on Wikipedia tables. Since the tables are represented as (subject, predicate, object) triples, it also suits our evaluation. Besides creating table-to-text annotations, the authors also use existing datasets: the QA dataset WikiSQL (Zhong et al., 2017), the cleaned E2E (Dušek et al., 2019) (entity-to-entity relations in the restaurant domain), and the original release of the WebNLG dataset for the 2017 challenge (Shimorina et al., 2018). The authors align the predicates such that predicates with the same meaning have the same representation. The dataset has 82K instances. We excluded the WikiSQL split as it has been generated automatically and after we performed a manual verification, we observed many low quality ground truth texts.

6.4 Evaluation
In this section, we consider (RQ2): Can we improve G2T generation on factual faithfulness?
Table 4 shows the results on the SimpleQuestions dataset. We generally have a high FactSpotter score, indicating that models are already good at relaying factual information. We can improve the factual faithfulness with F-T5 and FGT without significant compromise on other metrics, implying maintained fluency of texts.

<table><caption>Table 4: Results on G2T on SimpleQuestions. Here and below, T5S stands for T5 small, T5B for T5 base, F-T5S for FactT5 small, F-T5B for FactT5 base, JGT-T5 for JointGT-T5, and FGT-T5 for FactJointGT-T5.</caption><thead><tr><th>Model</th><th>BLEU</th><th>METEOR</th><th>BERTF1</th><th>BLEURT</th><th>BARTS</th><th>FactS</th></tr></thead><tbody><tr><td>T5S</td><td>37.97</td><td>36.50</td><td>93.43</td><td>67.85</td><td>-2.45</td><td>96.80</td></tr><tr><td>F-T5S</td><td>37.85</td><td>36.06</td><td>93.42</td><td>67.86</td><td>-2.45</td><td><strong>98.17</strong></td></tr><tr><td>T5B</td><td>38.73</td><td>36.43</td><td>93.61</td><td>68.48</td><td>-2.43</td><td>95.09</td></tr><tr><td>F-T5B</td><td>38.73</td><td>36.42</td><td>93.56</td><td>68.42</td><td>-2.43</td><td>97.14</td></tr><tr><td>JGT-T5</td><td><strong>39.35</strong></td><td><strong>36.82</strong></td><td><strong>93.65</strong></td><td><strong>68.50</strong></td><td><strong>-2.42</strong></td><td>95.40</td></tr><tr><td>FGT-T5</td><td>39.24</td><td>36.78</td><td>93.64</td><td>68.48</td><td><strong>-2.42</strong></td><td>97.25</td></tr></tbody></table>

<table><caption>Table 5: Results on G2T on WebNLG 2017 Const.</caption><thead><tr><th>Model</th><th>BLEU</th><th>METEOR</th><th>BERTF1</th><th>BLEURT</th><th>BARTS</th><th>FactS</th></tr></thead><tbody><tr><td>T5S</td><td>66.24</td><td>47.80</td><td>96.72</td><td>73.16</td><td>-1.41</td><td>98.67</td></tr><tr><td>F-T5S</td><td>66.27</td><td>47.89</td><td>96.73</td><td>73.21</td><td>-1.41</td><td>99.25</td></tr><tr><td>T5B</td><td>67.04</td><td><strong>48.35</strong></td><td><strong>96.81</strong></td><td>73.22</td><td>-1.40</td><td>99.44</td></tr><tr><td>F-T5B</td><td>67.04</td><td>48.22</td><td>96.80</td><td>73.26</td><td>-1.40</td><td><strong>99.71</strong></td></tr><tr><td>JGT-T5</td><td><strong>67.08</strong></td><td>48.34</td><td>96.76</td><td><strong>73.44</strong></td><td><strong>-1.39</strong></td><td>99.09</td></tr><tr><td>FGT-T5</td><td>66.89</td><td>48.19</td><td>96.84</td><td>73.42</td><td><strong>-1.39</strong></td><td>99.67</td></tr></tbody></table>

<table><caption>Table 6: Results on G2T on the WebNLG 2020 dataset.</caption><thead><tr><th>Model</th><th>BLEU</th><th>METEOR</th><th>BERTF1</th><th>BLEURT</th><th>BARTS</th><th>FactS</th></tr></thead><tbody><tr><td>T5S</td><td>52.30</td><td>40.82</td><td>93.43</td><td>-1.75</td><td>65.80</td><td>90.75</td></tr><tr><td>F-T5S</td><td>52.44</td><td>41.02</td><td>93.45</td><td>-1.74</td><td>65.92</td><td>93.45</td></tr><tr><td>T5B</td><td>54.29</td><td>41.66</td><td><strong>93.65</strong></td><td><strong>-1.69</strong></td><td>66.43</td><td>93.60</td></tr><tr><td>F-T5B</td><td><strong>54.72</strong></td><td><strong>41.70</strong></td><td>93.61</td><td><strong>-1.69</strong></td><td><strong>66.46</strong></td><td><strong>95.14</strong></td></tr><tr><td>JGT</td><td>54.23</td><td>41.49</td><td>93.47</td><td>-1.72</td><td>66.23</td><td>91.26</td></tr><tr><td>FGT-T5</td><td>54.45</td><td>41.52</td><td>93.49</td><td>-1.72</td><td>66.31</td><td>93.16</td></tr></tbody></table>

<table><caption>Table 7: Results on G2T on the DART dataset.</caption><thead><tr><th>Model</th><th>BLEU</th><th>METEOR</th><th>BERTF1</th><th>BLEURT</th><th>BARTS</th><th>FactS</th></tr></thead><tbody><tr><td>T5S</td><td>46.22</td><td>39.96</td><td>94.69</td><td>66.62</td><td>-2.03</td><td>95.47</td></tr><tr><td>F-T5S</td><td>46.31</td><td>40.07</td><td>94.74</td><td>66.66</td><td>-2.02</td><td>97.29</td></tr><tr><td>T5B</td><td><strong>48.47</strong></td><td><strong>40.74</strong></td><td>95.04</td><td><strong>67.49</strong></td><td><strong>-1.97</strong></td><td>96.65</td></tr><tr><td>F-T5B</td><td>48.37</td><td>40.72</td><td><strong>95.05</strong></td><td>67.43</td><td><strong>-1.97</strong></td><td><strong>97.60</strong></td></tr><tr><td>JGT-T5</td><td>47.51</td><td>40.43</td><td>94.92</td><td>67.33</td><td>-2.01</td><td>95.86</td></tr><tr><td>FGT-T5</td><td>47.39</td><td>40.32</td><td>94.92</td><td>67.26</td><td>-2.00</td><td>97.25</td></tr></tbody></table>

<footnote n="2">²https://dki-lab.github.io/GrailQA/</footnote>

<footnote n="3">³https://gitlab.com/shimorina/webnlg-dataset</footnote>
<table><caption>Table 8: Results on G2T on the GrailQA dataset.</caption><thead><tr><th>Model/Split</th><th>BLEU</th><th>METEOR</th><th>BERTF1</th><th>BLEURT</th><th>BARTS</th><th>FactS</th></tr></thead><tbody><tr><td><strong>IID</strong></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>T5S</td><td>44.51</td><td>41.80</td><td>93.23</td><td>69.53</td><td>-2.37</td><td>97.98</td></tr><tr><td>F-T5S</td><td>44.64</td><td>41.88</td><td>93.25</td><td>69.63</td><td>-2.36</td><td>98.47</td></tr><tr><td>T5B</td><td>45.95</td><td>42.71</td><td>93.50</td><td>70.66</td><td><strong>-2.29</strong></td><td>99.43</td></tr><tr><td>F-T5B</td><td><strong>46.10</strong></td><td><strong>42.67</strong></td><td><strong>93.52</strong></td><td><strong>70.73</strong></td><td><strong>-2.29</strong></td><td><strong>99.50</strong></td></tr><tr><td>JGT-T5</td><td>43.68</td><td>41.65</td><td>93.21</td><td>69.41</td><td>-2.37</td><td>98.62</td></tr><tr><td>FGT-T5</td><td>43.61</td><td>41.65</td><td>93.19</td><td>69.41</td><td>-2.37</td><td>99.12</td></tr><tr><td><strong>Zero</strong></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>T5S</td><td>30.30</td><td>36.91</td><td>91.74</td><td>62.87</td><td>-2.74</td><td>93.27</td></tr><tr><td>F-T5S</td><td>30.30</td><td>36.90</td><td>91.75</td><td>63.01</td><td>-2.73</td><td>94.60</td></tr><tr><td>T5B</td><td>32.20</td><td>37.35</td><td>91.92</td><td>63.84</td><td>-2.72</td><td>94.77</td></tr><tr><td>F-T5B</td><td>32.39</td><td>37.39</td><td>91.92</td><td>63.94</td><td>-2.71</td><td><strong>95.61</strong></td></tr><tr><td>JGT-T5</td><td><strong>32.94</strong></td><td><strong>37.69</strong></td><td><strong>92.02</strong></td><td><strong>64.18</strong></td><td><strong>-2.68</strong></td><td>94.15</td></tr><tr><td>FGT-T5</td><td>32.46</td><td>37.55</td><td>91.93</td><td>64.00</td><td><strong>-2.68</strong></td><td>94.95</td></tr><tr><td><strong>Comp.</strong></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>T5S</td><td>30.38</td><td>35.32</td><td>92.09</td><td>63.99</td><td>-2.74</td><td>94.94</td></tr><tr><td>F-T5S</td><td>30.14</td><td>35.21</td><td>92.08</td><td>63.72</td><td>-2.75</td><td>96.58</td></tr><tr><td>T5B</td><td><strong>31.75</strong></td><td>35.64</td><td>92.24</td><td>63.99</td><td>-2.72</td><td>94.84</td></tr><tr><td>F-T5B</td><td>31.66</td><td><strong>35.72</strong></td><td>92.24</td><td>64.10</td><td>-2.71</td><td>96.53</td></tr><tr><td>JGT-T5</td><td>31.46</td><td>36.08</td><td>92.39</td><td>64.92</td><td>-2.67</td><td>95.26</td></tr><tr><td>FGT-T5</td><td>31.25</td><td>36.21</td><td><strong>92.43</strong></td><td><strong>65.12</strong></td><td><strong>-2.65</strong></td><td><strong>97.10</strong></td></tr></tbody></table>

Table 5 has the highest FactSpotter score from all datasets, which means that we observe the most factual generations on WebNLG 2017, with F-T5 and FGT having slightly higher scores.

In Table 6, the FactSpotter scores are lower for the WebNLG 2020 test split, although we achieve scores comparable to WebNLG 2017 on its validation split. This discrepancy may be attributed to the difference in distribution between the test and training splits of WebNLG 2020. F-T5B can achieve higher FactSpotter score than T5B without compromising fluency. We observe the same trends for the DART dataset in Table 7.

In Table 8, all the metrics are higher for the IID split of GrailQA, and in particular FactSpotter can reach 99.5%, hence the models learn to reproduce triples seen in training. For Zero-shot and Compositional splits, larger models are better, and our factual inference improves the score of FactSpotter. We illustrate some improved samples of Zero-shot and Compositional splits in Appendix A.3, and in Appendix A.6, we investigate the impact of varying numbers of triples in the input subgraphs on the quality of the generated text.

To validate that indeed generations improve using FactSpotter in inference, we select the best FGT-T5 model and we analyse the top 20 phrases where the FactSpotter score improved the most compared to the JGT-T5 generations.

For the SimpleQuestion datasets, we have 15 generations that are more factual, and 5 generations less factual. For the Zero-shot split of GrailQA, 14 generations are more factual. For its Composi-

tional split, we have 13 improved generations, but also 5 that are less factual. For the IID split, 4 generations are improved, others are all rephrased texts. Only 7 samples of IID improve over 0.01 for FactSpotter, so this split is not challenging. For the DART dataset, 6 texts are more factual. DART dataset has samples that ground-truth sentences do not match with graphs, so FactSpotter trained on DART has false positives. For WebNLG 2017 dataset, 11 generations are more factual, others are rephrased texts. WebNLG 2017 is only has 16 generations improve over 0.1 for FactSpotter, whose baseline is very high. For WebNLG 2020, 12 generations are more factual, and 3 are rephrased texts. 5 generations in WebNLG 2020 have higher FactSpotter than baseline generations, but they're still not factual enough.

For the cases where a generation becomes less factual, this is a consequence of the accuracy of FactSpotter, which we present in Section 4. Given that our metric does not correlate strongly with fluency, we perform a second analysis on generations to observe if there is a decrease in fluency. To answer this question, we study the top 20 sentences for which the BLEURT decreased the most in comparison with the original generated question. We do not observe an obvious decrease in fluency on any dataset, the decrease in BLEURT score is due to several other factors: BLEURT has difficulties identifying rephrased sentences, in a few cases the factual faithfulness decreased, and in the remaining cases the generations are more faithful to the input graph than the ground truth sentences, however BLEURT cannot identify it. Hence, we can conclude that adding FactSpotter as a plugin in generation can improve G2T generations on factual faithfulness and does not affect the fluency.

## 7 Remaining challenges in G2T task

Finally, we consider (RQ3): is G2T task solved on existing datasets? We have observed high FactSpotter scores in Section 6 on the performances of models. We use FactSpotter to do a more detailed analysis: we investigate what is the percentage of generations in each dataset which had at least a fact considered missing by FactSpotter. A fact is considered as missing if the score of the pair (fact, generated sentence) is lower than 0.5. We obtain the following statistics: 1.94% of texts miss a fact in SimpleQuestions; 7.27% of texts miss at least a fact in DART; 5.79% of WebNLG 2017 texts
miss at least one fact, and 12.64% for WebNLG 2020; For GrailQA we have 5.8% for the zero shot split, 4.36% for the compositional split and 1.13 for the IID split. According to the observations, WebNLG 2020 is the most challenging dataset, followed by DART, the zero shot split of GrailQA, and the WebNLG 2017 dataset. In Appendix A.7, taking GrailQA and WebNLG 2017 as examples, we analysed the difficulty of G2T on datasets from different knowledge graphs, by looking into how often predicates and entity names are rephrased or expressed exactly as in the input graph.

We perform a second evaluation, this time by manually analyzing the output of the models. We consider the worst 20 sentences according to BLEURT and FactSpotter, hence 40 examples per dataset or split. On SimpleQuestions, the generations are fluent, however 22/40 have an incorrect predicate. For GrailQA, in the IID split the predicates are correctly generated, but the models still have difficulties in generating some entity names (16/40). For the zero-shot split, generations suffer from wrong entities and predicates (22/40). The compositional split has several samples with wrong ground truth (6/40 of the worst generations) and 19 out of 40 incorrect generations. For DART, 24 generations are not correct. On WebNLG 2017, from the worst 40 generations, only two might benefit from improved fluency, while in many examples, the generated sentence was more fluent than the ground truth (14/40). Regarding correctness, only 2 out of 40 generations had a missing triple, while two generations incorrectly used a predicate. On WebNLG2020, only one instance exhibits room for improvement in fluency, but 24 instances either omit factual information or contain incorrect facts. Among the 20 outputs with the lowest BLEURT scores, 9 are rephrased texts with correctly explained facts. In contrast, among the 20 outputs with the lowest FactSpotter, only 4 instances fall into this category.

Based on our manual annotations, we observed that models are able to produce correct generations. However, when the generation is rephrased in respect to the ground truth sentence, metrics like BLEURT, which measure if two sentences are equivalent, struggle to assign high scores. We recall that BLEURT, a normalized metric, gives a score of 1 to equivalent sentences. On the dataset WebNLG 2017, our metric assigns a very high score to the models, while the highest average BLEURT score

is 73.44%. The BLEURT scores of the generations vary from 0.46 to 0.99; more than 50% of the test set generations score less than 0.80. We sampled 40 generations with BLEURT score < 0.8 and note that 35 generations are correct, which are rephrasing the ground truth, while 2 out of 35 that are better than the ground truth. Hence, we observe that BLEURT score cannot be used to determine if we have achieved a good performance on a dataset, it can only be used to compare different models. This issue has also been pointed out by the authors<fnref n="4" />.

FactSpotter answers whether a fact is present in text; it does not have to address the much harder task of deciding if two sentences are equivalent. Besides being more reliable because it is solving a simpler task, it is also more interpretable as we can investigate the exact triples that are classified as negative, instead of doing a complete comparison between a subgraph and a sentences or between two sentences. This is especially useful for long input graphs and long generations.

## 8 Conclusion

In this work, we have presented a new metric for measuring factual faithfulness in G2T, FactSpotter. This metric can be trained in a self supervised fashion, using the same annotations as a G2T model. We have shown it achieves the highest correlation with humans on factual faithfulness and it can be used as a plug-in feature for G2T models. Finally, we have used our metric to analyze the difficulty of existing datasets. We have observed that models perform very well on these datasets, hence new datasets should be proposed. Such datasets could cover more difficult input graphs, for example triple from tables. In addition, through the initiative of governments to release tabular data related to public interest⁵, tools trained to express in natural language the content of tables could be used as user friendly interfaces for citizens.

**Acknowledgment.** This work was performed using HPC resources from GENCI-IDRIS (Grant 2023-AD011014244). The authors were partially funded by the ANR-20-CHIA-0015 project.

<footnote n="4">⁴https://github.com/google-research/bleurt/issues/1

⁵For example, Eurostat, https://ec.europa.eu</footnote>
# 9 Limitations

Our work has the following limitations:

* FactSpotter cannot be used to determine the precise nature of the error in the generated sentence. It was trained to predict whether a fact is presented in the text or not, not if we the sentence has a wrong predicate or a wrong subject or object. This problem can be solved by a second classification step for whether predicates or entities are incorrectly verbalized in the text, to make FactSpotter more interpretable.

* The input of FactSpotter is the concatenation of a fact *f* represented in triple and a natural language text *T*, i.e., it has limited input format. With such input, the advantage is that it is easier to construct both positive and negative samples for its self-supervised training. However, it is also difficult to use it to check the factual faithfulness on other text generation tasks, since high quality structured knowledge graphs are hard to generate. However, we will investigate in the future the use of open information extraction models (Upadhyay et al., 2023) for extracting facts from sentences.

* Although the accuracy and F1 score of our classification model on the test splits of various datasets in Table 1 are high, there still exist some false positive and false negative samples. Hence, our FactSpotter generally reflects factual faithfulness, but it might still be biased on some hard samples, especially when predicates in knowledge graphs are distant to their natural language representations in the vector space of language models.

# References

Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. Dbpedia: A nucleus for a web of open data. In *The Semantic Web: 6th International Semantic Web Conference, 2nd Asian Semantic Web Conference, ISWC 2007+ ASWC 2007, Busan, Korea, November 11-15, 2007. Proceedings*, pages 722–735. Springer.

Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In *Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization*, pages 65–72.

Anja Belz and Ehud Reiter. 2006. Comparing automatic and human evaluation of NLG systems. In *11th Conference of the European Chapter of the Association for Computational Linguistics*, pages 313–320, Trento, Italy. Association for Computational Linguistics.

Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: A collaboratively created graph database for structuring human knowledge. In *Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data*, SIGMOD '08, page 1247–1250, New York, NY, USA. Association for Computing Machinery.

Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. 2015. Large-scale simple question answering with memory networks. *arXiv preprint arXiv:1506.02075*.

Thiago Castro Ferreira, Claire Gardent, Nikolai Ilinykh, Chris van der Lee, Simon Mille, Diego Moussallem, and Anastasia Shimorina. 2020. The 2020 bilingual, bi-directional WebNLG+ shared task: Overview and evaluation results (WebNLG+ 2020). In *Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+)*, pages 55–76, Dublin, Ireland (Virtual). Association for Computational Linguistics.

Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: A survey. *arXiv preprint arXiv:2006.14799*.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In *International conference on machine learning*, pages 1597–1607. PMLR.

Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. *arXiv preprint arXiv:2003.10555*.

Pierre Jean A Colombo, Chloé Clavel, and Pablo Pi-antanida. 2022. InfoLM: A new metric to evaluate summarization & data2text generation. In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 36, pages 10554–10562.

Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William Cohen. 2019. Handling divergent reference texts when evaluating table-to-text generation. In *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pages 4884–4895, Florence, Italy. Association for Computational Linguistics.

Ondřej Dušek, David M. Howcroft, and Verena Rieser. 2019. Semantic noise matters for neural natural language generation. In *Proceedings of the 12th International Conference on Natural Language Generation*, pages 421–426, Tokyo, Japan. Association for Computational Linguistics.
James D Evans. 1996. *Straightforward statistics for the behavioral sciences*. Thomson Brooks/Cole Publishing Co.

Juliette Faille, Albert Gatt, and Claire Gardent. 2021. Entity-based semantic adequacy for data-to-text generation. In *Findings of the Association for Computational Linguistics: EMNLP 2021*, pages 1530–1540, Punta Cana, Dominican Republic. Association for Computational Linguistics.

Angela Fan, Claire Gardent, Chloé Braud, and Antoine Bordes. 2019. Using local knowledge graph construction to scale Seq2Seq models to multi-document inputs. In *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*, pages 4186–4196, Hong Kong, China. Association for Computational Linguistics.

Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. Creating training corpora for NLG micro-planners. In *Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 179–188, Vancouver, Canada. Association for Computational Linguistics.

Yu Gu, Sue Kase, Michelle Vanni, Brian Sadler, Percy Liang, Xifeng Yan, and Yu Su. 2021. Beyond i.i.d.: Three levels of generalization for question answering on knowledge bases. In *Proceedings of the Web Conference 2021*, WWW ’21, page 3477–3488, New York, NY, USA. Association for Computing Machinery.

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. *ACM Comput. Surv.*, 55(12).

Pei Ke, Haozhe Ji, Yu Ran, Xin Cui, Liwei Wang, Lin-
feng Song, Xiaoyan Zhu, and Minlie Huang. 2021.
JointGT: Graph-text joint representation learning for
text generation from knowledge graphs. In *Find-
ings of the Association for Computational Linguis-
tics: ACL-IJCNLP 2021*, pages 2526–2538, Online.
Association for Computational Linguistics.

Moniba Keymanesh, Adrian Benton, and Mark Dredze. 2022. What makes data-to-text generation hard for pretrained language models? In *Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM)*, pages 539–554, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Klaus Krippendorff. 2018. *Content analysis: An intro-
duction to its methodology*. Sage publications.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.

BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 7871–7880, Online. Association for Computational Linguistics.

Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In *Text summarization branches out*, pages 74–81.

Tianyu Liu, Yizhe Zhang, Chris Brockett, Yi Mao, Zhifang Sui, Weizhu Chen, and Bill Dolan. 2022. A token-level reference-free hallucination detection benchmark for free-form text generation. In *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 6723–6737, Dublin, Ireland. Association for Computational Linguistics.

Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, Yangxiaokang Liu, Nadia Irwanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Mutethia Mutuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Chern Tan, Xi Victoria Lin, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. 2021. DART: Open-domain structured data record to text generation. In *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 432–447, Online. Association for Computational Linguistics.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-
Jing Zhu. 2002. BLEU: a method for automatic eval-
uation of machine translation. In *Proceedings of the
40th annual meeting of the Association for Computa-
tional Linguistics*, pages 311–318.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. *OpenAI blog*, 1(8):9.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. *The Journal of Machine Learning Research*, 21(1):5485–5551.

Clement Rebuffel, Thomas Scialom, Laure Soulier, Benjamin Piwowarski, Sylvain Lamprier, Jacopo Staiano, Geoffrey Scoutheeten, and Patrick Gallinari. 2021. Data-questeval: A referenceless metric for data-to-text semantic evaluation. In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pages 8029–8036.

Leonardo F. R. Ribeiro, Martin Schmitt, Hinrich Schütze, and Iryna Gurevych. 2021. Investigating pretrained language models for graph-to-text generation. In *Proceedings of the 3rd Workshop on Natural Language Processing*, pages 8029–8036.
*Language Processing for Conversational AI*, pages 211–227, Online. Association for Computational Linguistics.

Leonardo F. R. Ribeiro, Yue Zhang, Claire Gardent, and Iryna Gurevych. 2020. Modeling global and local node contexts for text generation from knowledge graphs. *Transactions of the Association for Computational Linguistics*, 8:589–604.

Julien Romero and Simon Razniewski. 2020. Inside quasimodo: Exploring construction and usage of commonsense knowledge. In *Proceedings of the 29th ACM International Conference on Information & Knowledge Management*, pages 3445–3448.

Ananya B Sai, Akash Kumar Mohankumar, and Mitesh M Khapra. 2022. A survey of evaluation metrics used for nlg systems. *ACM Computing Surveys (CSUR)*, 55(2):1–39.

Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. BLEURT: Learning robust metrics for text generation. *arXiv preprint arXiv:2004.04696*.

Anastasia Shimorina, Claire Gardent, Shashi Narayan, and Laura Perez-Beltrachini. 2018. WebNLG challenge: Human evaluation results. Ph.D. thesis, Loria & Inria Grand Est.

Prajna Upadhyay, Oana Balalau, and Ioana Manolescu. 2023. Open information extraction with entity focused constraints. In *Findings of the Association for Computational Linguistics: EACL 2023*, pages 1285–1296, Dubrovnik, Croatia. Association for Computational Linguistics.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. *Advances in neural information processing systems*, 30.

Rand R Wilcox. 2016. Comparing dependent robust correlations. *British Journal of Mathematical and Statistical Psychology*, 69(3):215–224.

Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015. Semantic parsing via staged query graph generation: Question answering with knowledge base. In *Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)*, pages 1321–1331, Beijing, China. Association for Computational Linguistics.

Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. *Advances in Neural Information Processing Systems*, 34:27263–27277.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. *arXiv preprint arXiv:1904.09675*.

Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. *arXiv preprint arXiv:1709.00103*.

Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. 2018. Commonsense knowledge aware conversation generation with graph attention. In *IJCAI*, pages 4623–4629.

# A Appendix

## A.1 Metrics for the evaluation of G2T

**BLEU (Papineni et al., 2002)** This is one of the oldest sentence similarity metrics. BLEU is computed as a weighted geometric mean of *n*-gram precision scores, that is: it rewards inputs that share many common substrings (or *n*-grams, usually *n* is 4). The score ranges between 0 and 1.

**METEOR (Banerjee and Lavie, 2005)** This similarity metric is based on the harmonic mean of the 1-gram precision and recall. Its recall has a higher weight than precision. METEOR scores range between 0 and 1.

**BERTScore (Zhang et al., 2019)** This leverages the pre-trained contextual embeddings of BERT, and matches words in candidate and reference sentences by cosine similarity. The score can be used to compute precision, recall, and F1. BERTScore is also a similarity measure, ranging from 0 to 1.

**BLEURT (Sellam et al., 2020)** This is a fine-tuned Bert model on synthetically generated texts. It provides a similarity score learned from evaluation scores such as BLEU or ROUGE, and from human ratings; 1 represents a perfect match, while a value close to 0 means no similarity. We consider the authors’ latest released model, BleuRT-20⁶.

**BARTScore (Yuan et al., 2021)** This metric is based on the BART seq2seq language model: the score represents the weighted log probability of one text *y*, given another text *x*. We compute two versions of the BARTScore, based on the indications of the paper: in the first version of the score, we compare the generated sentence with the gold standard (we refer to it as BARTScore, the default version provided by the authors), and a second version in which we compare the generated text with the input graph (referred in the paper as the faithfulness method, BARTScore-faithful in our experiments). The second score should encourage

<footnote n="6">⁶https://github.com/google-research/bleurt</footnote>
the metric to consider the factual faithfulness of a
generation. BARTScore is not normalized.

A.2 Bootstrapping

Given two metrics $M_1, M_2$, we need to assess
which one better correlates with human annota-
tions. We have at our disposal a set of G2T systems
$S_1, \dots, S_S$, and a set $A$ of G2T samples. Given
a set $D$ of dimensions, e.g., fluency, factfulness,
etc., we have at our disposal the human annota-
tion $H^d(S_i(a)), d \in D \ 1 \le i \le S, a \in A$: this
is the score that human users assigned to the G2T
generation of system $S_i$, on the sample $a$, along
dimension $d$. Similarly, each automated metric $M$
computes $M^d(S_i(a))$, the $M$ score of $S_i$ on sam-
ple $a$ along dimension $d$. We note however that
automated metric give generally just one score for
all dimensions.

Based on the $H^d(S_i(a)), M^d(S_i(a))$ over all
systems, samples, and dimensions, we assess how
well metric $M$ correlates with human judgment $H$
in a statistically sound way. Since each dimension
is of independent interest, we study the correlation
of $M$ and $H$ separately for each dimension, and
omit $d$ from the notations below, for simplicity.

We compute the three correlation coefficients ($r$, $\rho$ and $\tau$) between vectors of per-system scores $[M(S_1), \dots, M(S_S)]$ and $[H(S_1), \dots, H(S_S)]$. One aspect remains to settle: for a given $S_i$, how to aggregate information from all the annotated samples $A$? It is important to do this in a way that is robust to the divergence and noise sometimes found in human annotations.

As is proposed in (Dhingra et al., 2019), we
apply bootstrapping, as follows. We generate a
number (we used 1000, which is same as (Dhingra
et al., 2019)) of "variants" $A_1, \dots, A_{1000}$ of the an-
notated samples $A$, by sampling with replacement
from $A$. Each $A_i$ has the same size as $A$, but it
may contain some $A$ samples several times, and
others not at all. We then compute correlation coef-
ficients between the vectors $[M(S_1), \dots, M(S_S)]$
and $[H(S_1), \dots, H(S_S)]$ over each $A_i$, then repre-
sent each of the resulting 1000 correlation values
by their mean, together a confidence interval. The
relatively large number of "variants" acts as insula-
tion against divergence and noise in the annotations.
Then for two metrics $M_1, M_2$:

i) If the mean of $M_1$ is higher than $M_2$ and their
confidence intervals do not overlap, it means $M_1$
better correlates with human judgment, than $M_2$.

ii) If the confidence intervals of $M_1, M_2$ overlap,
we need to check whether a statistically significant
conclusion can be drawn about the metrics. This is
the case only if the confidence interval of the dif-
ference in correlation between the metrics is above
0 (Wilcox, 2016), in which the metric with higher
mean correlates better with human judgment.

A.3 Improved Samples of G2T Generation

Table 9 shows the improvement of factual faith-
fulness in G2T generation task after integrating
*FactSpotter* into G2T inference. The samples are
from the hardest zero-shot and compositional splits
of GrailQA dataset. Each sample in the table in-
cludes a description of a subgraph, a ground truth
text, a text generated by the best baseline model
(JointGT), and a text generated by our FactJointGT.

Although the texts generated by the baseline
model, JointGT, are generally fluent, we observe
hallucinations in its generations: i) Some predi-
cates are interpreted with different meanings in
the generated texts; ii) Some entities are generated
wrongly; iii) Some facts in subgraphs are lost in the
generated texts. After integrating FactSpotter into
the inference of G2T generation, without retraining
the G2T model, our FactJointGT can generate texts
that verbalize facts in subgraphs more correctly
and completely. Hence the factual faithfulness of
G2T task is improved by FactSpotter, but the other
metrics do not necessary improve, since rephrased
high quality texts might also be punished.

A.4 Parameter tuning

A.4.1 Percentages in negative samples for
FactSpotter **training**

FactSpotter receives in the training set two types
of negative samples. In the first type of negative
sample (type I) we modify the fact from the graph,
while the second type of negative sample (type II)
we modify the text. To determine the best balance
between negative samples of type I, respectively,
type II, we compute the correlations between the
score of FactSpotter and a subset of 50 annotations
per system from the 2020 WebNLG challenge. As
shown in Table 10, the best results for detecting
factual accuracy and fluency are using either a ratio
0.9-0.1 or 0.8-0.2. In the remaining of the paper,
we use FactSpotter with the parameters 0.9-0.1.

A.4.2 λ tuning for factual generations

We recall that in Equation 1, we have defined a
weight $\lambda$ that quantifies the importance that we
<table><caption>Table 9: Texts with improved factual faithfulness after integrating FactSpotter into G2T inference. Red stands for the contents that are presented in subgraphs, but are missed in the baseline generations. Blue stands for the exact contents correctly presented in ground-truth texts or FactJointGT generations.</caption><thead><tr><th>Subgraph</th><th>Ground Truth</th><th>JointGT Baseline</th><th>FactJointGT</th></tr></thead><tbody><tr><td>(spacecraft, manufacturer, International Telephone & Telegraph), (spacecraft, manufacturer, Boeing Company), (spacecraft, count, )</td><td>What is the total number of spacecrafts manufactured by International Telephone & Telegraph and Boeing Company?</td><td>International Telephone & Telegraph manufactures how many spacecrafts?</td><td>How many spacecrafts did International Telephone & Telegraph manufacture with Boeing Company?</td></tr><tr><td>(music recording, artist, Celine Dion), (music recording, length, float), (float, argmin, )</td><td>What is the shortest Celine Dion song?</td><td>What is the musical recording that was recorded by Celine Dion?</td><td>Which musical recording was recorded by Celine Dion with the shortest length?</td></tr><tr><td>(John Elliott, games published, computer game expansion), (computer game expansion, count, )</td><td>How many game expansions has John Elliott released?</td><td>How many video game expansions were published by John Ellison?</td><td>How many video game expansions were published by John Elliott?</td></tr><tr><td>(conference series, conferences of this type, seminar), (conference series, sponsoring organization, Australian Centre for Psychoanalysis), (conference series, question)</td><td>What conference series that has a sponsoring organization of Australian Centre for Psychoanalysis and is of type seminar?</td><td>Australian Centre for Psychoanalysis sponsors which conference series?</td><td>What conference series is sponsored by Australian Centre for Psychoanalysis and is a seminar?</td></tr><tr><td>(railway, terminuses, Strabane), (railway, question)</td><td>Strabane is the terminuses of which railway?</td><td>Which railway has a terminus of Stratane?</td><td>Strabane is the terminus of which railway?</td></tr></tbody></table>

<table><caption>Table 10: The influence of the ratio on FactSpotter.</caption><thead><tr><th rowspan="2">Type I/<br>Type II</th><th colspan="3">Data Coverage</th><th colspan="3">Relevance</th><th colspan="3">Fluency</th></tr><tr><th>r</th><th>ρ</th><th>τ</th><th>r</th><th>ρ</th><th>τ</th><th>r</th><th>ρ</th><th>τ</th></tr></thead><tbody><tr><td>0.9 / 0.1</td><td>0.85</td><td>0.84</td><td>0.68</td><td>0.93</td><td>0.88</td><td>0.75</td><td>0.91</td><td>0.79</td><td>0.63</td></tr><tr><td>0.8 / 0.2</td><td>0.83</td><td>0.85</td><td>0.63</td><td>0.92</td><td>0.93</td><td>0.79</td><td>0.89</td><td>0.79</td><td>0.61</td></tr><tr><td>0.7 / 0.3</td><td>0.77</td><td>0.84</td><td>0.66</td><td>0.85</td><td>0.89</td><td>0.73</td><td>0.66</td><td>0.63</td><td>0.50</td></tr></tbody></table>

<table><caption>Table 11: BleuRT vs. FactSpotter when varying the λ parameter (importance of the fact classifier for generating a sentence, see Equation 1).</caption><thead><tr><th rowspan="2">λ</th><th colspan="2">IID</th><th colspan="2">Zero-shot</th><th colspan="2">Compositional</th></tr><tr><th>BleuRT</th><th>FactS</th><th>BleuRT</th><th>FactS</th><th>BleuRT</th><th>FactS</th></tr></thead><tbody><tr><td>0.0</td><td>69.53</td><td>97.98</td><td>62.27</td><td>93.27</td><td><strong>63.99</strong></td><td>94.94</td></tr><tr><td>0.05</td><td><strong>69.76</strong></td><td>98.15</td><td>62.94</td><td>94.15</td><td>63.92</td><td>95.69</td></tr><tr><td>0.1</td><td><strong>69.76</strong></td><td><strong>98.48</strong></td><td>63.00</td><td>94.56</td><td>63.71</td><td>95.83</td></tr><tr><td>0.15</td><td>69.63</td><td>98.47</td><td><strong>63.01</strong></td><td>94.60</td><td>63.72</td><td><strong>96.58</strong></td></tr><tr><td>0.2</td><td>69.47</td><td>98.38</td><td>62.95</td><td><strong>94.79</strong></td><td>63.61</td><td>96.19</td></tr></tbody></table>

assign to the probability that the next word is predicted such that we increase the probability of having a fact in the generated text, versus the default probability of choosing a word based on previously generated tokens. In Table 11, we vary the value

of λ on the different splits of GrailQA on the T5 small model. The BleuRT score does not vary significantly, meaning that the fluency of the G2T model should not be affected. We fix λ to 0.15 for the experimental evaluation in Section 6, as it gives the best performance on the zero shot split, which is the most challenging, and gives very good results also for the other splits. More precisely, we report the results of FactT5 and FactJointGT with λ = 0.15 on the datasets introduced in Section 6.3.

## A.5 Sentence level correlation

We explain two distinct definitions of sentence-level correlation provided by literature and report the correlations on the WebNLG human annotations.

The first definition of sentence-level correlation (Colombo et al., 2022) is outlined as follows:

* Construct a pair containing the automated metric scores [$M(\text{sys}_1), \dots, M(\text{sys}_S)]$ and the corresponding system-level human scores [$H(\text{sys}_1), \dots, H(\text{sys}_S)]$ for a given sentence.
* The Pearson correlation amidst these pairs is computed. If the correlation is significant (adopting *p* < 0.05 as the threshold), this correlation is preserved.

* Then, report the average over all significant correlations (average over at most the total number of annotated sentences).

When there is a large number of significant pairs with high correlation value, it can show if a metric can be used to compare different verbalisations of the same input triples. We report the human correlation results computed as this definition on the WebNLG 2017 and 2020 annotations in Tables 12 and 13 respectively. We note that only *FactSpotter* is significantly larger than the other metrics on data coverage and relevance for WebNLG 2020, and semantic adequacy for WebNLG 2017, which are metrics about factual faithfulness. Please note that here it is important to report on how many sentences the correlation was significant with *p* < 0.05. Because we compute if a metric has a higher score than a second metric using the bootstrapping technique presented in the Annexes, different samples might have different number of sentences with correlation having *p* < 0.05, hence we report the average number of sentences over all the samples. For WebNLG2017, out of 223 human annotated sentences, we have for semantics: BLEURT 171, FactSpotter 172, BERT 121 and BART 169. For fluency we have BLEURT 80 sentence pairs, FactSpotter 48, BERT 71 and BART 55. For textual structure BLEURT had 104 sentences pairs, FactSpotter 65, BERT 94 and BART 75. We note that for semantics the majority of pairs are significant (172 out of 223), meaning that FactSpotter can predict with high correlation improved verbalisations of the same input graph, hence it can be used to solve Q2. For WebNL2017, out of 178 human annotated sentences, we have: i) for correctness, BLEURT 121, FactSpotter 83, BERT 95 and BART 97 significant pairs on average; ii) for data coverage, BLEURT 102, FactSpotter 100, BERT 65 and BART 106; iii) for fluency, BLEURT 105, FactSpotter 49, BERT 90 and BART 72; iv) for relevance, BLEURT 99, FactSpotter 67, BERT 71 and BART 78; v) for text structure, BLEURT 92, FactSpotter 42, BERT 75 and BART 59; In this dataset, we obtain far less significant pairs, but we will show next that the annotators had more disagreement on WebNLG 2020.

<table><caption>Table 12: The result of sentence-level correlation with the first definition on the WebNLG 2017 annotation.</caption><thead><tr><th>Metrics</th><th>Sem. Adeq.</th><th>T. Structure</th><th>Fluency</th></tr></thead><tbody><tr><td>Human</td><td>0.80</td><td>0.77</td><td>0.77</td></tr><tr><td>BertF1</td><td>0.80</td><td>0.78</td><td>0.75</td></tr><tr><td>BleuRT</td><td>0.82</td><td>0.79</td><td>0.78</td></tr><tr><td>BartS</td><td>0.81</td><td>0.77</td><td>0.78</td></tr><tr><td>FactS</td><td><strong>0.84</strong></td><td>0.78</td><td>0.78</td></tr></tbody></table>

<table><caption>Table 13: The result of sentence-level correlation with the first definition on the WebNLG 2020 annotation.</caption><thead><tr><th>Metrics</th><th>Correct.</th><th>D. Cover.</th><th>Fluency</th><th>Relev.</th><th>T. Struct.</th></tr></thead><tbody><tr><td>Human</td><td>0.67</td><td>0.68</td><td>0.62</td><td>0.65</td><td>0.67</td></tr><tr><td>BertF1</td><td>0.65</td><td>0.67</td><td>0.64</td><td>0.66</td><td>0.66</td></tr><tr><td>BleuRT</td><td><strong>0.72</strong></td><td>0.68</td><td><strong>0.68</strong></td><td>0.71</td><td><strong>0.70</strong></td></tr><tr><td>BartS</td><td>0.71</td><td>0.69</td><td>0.66</td><td>0.70</td><td>0.68</td></tr><tr><td>FactS</td><td>0.69</td><td><strong>0.71</strong></td><td>0.50</td><td><strong>0.71</strong></td><td>0.60</td></tr></tbody></table>

<table><caption>Table 14: The result of sentence-level correlation with the second definition on the WebNLG 2017 annotation.</caption><thead><tr><th>Metrics</th><th>Sem. Adeq.</th><th>T. Structure</th><th>Fluency</th></tr></thead><tbody><tr><td>Human</td><td>0.56</td><td>0.44</td><td>0.50</td></tr><tr><td>BertF1</td><td>0.64</td><td>0.54</td><td>0.57</td></tr><tr><td>BleuRT</td><td><strong>0.69</strong></td><td><strong>0.55</strong></td><td><strong>0.59</strong></td></tr><tr><td>BartS</td><td>0.59</td><td>0.47</td><td>0.45</td></tr><tr><td>FactS</td><td>0.65</td><td>0.43</td><td>0.43</td></tr></tbody></table>

<table><caption>Table 15: The result of sentence-level correlation with the second definition on the WebNLG 2020 annotation.</caption><thead><tr><th>Metrics</th><th>Correct.</th><th>D. Cover.</th><th>Fluency</th><th>Relev.</th><th>T. Structure</th></tr></thead><tbody><tr><td>Human</td><td>0.38</td><td>0.37</td><td>0.29</td><td>0.35</td><td>0.30</td></tr><tr><td>BertF1</td><td>0.42</td><td>0.38</td><td>0.42</td><td>0.37</td><td>0.40</td></tr><tr><td>BleuRT</td><td><strong>0.45</strong></td><td><strong>0.43</strong></td><td><strong>0.47</strong></td><td><strong>0.38</strong></td><td><strong>0.43</strong></td></tr><tr><td>BartS</td><td>0.39</td><td>0.38</td><td>0.32</td><td>0.35</td><td>0.31</td></tr><tr><td>FactS</td><td>0.38</td><td>0.40</td><td>0.29</td><td>0.37</td><td>0.27</td></tr></tbody></table>

The second definition of sentence-level correlation (Banerjee and Lavie, 2005) is computed between the vector containing all the automatic scores for each sentence by a system *S* given by a metric *M*, and the vector containing the human metrics for each sentence. The sentence-level correlation of a metric *M* is computed as an average over all the correlation of each system *S*₁, ..., *S*ₙ. This measure determines if a metric can be used to rank verbalisations of different input graphs. The human correlation results computed as the second definition on the WebNLG 2017 and 2020 annotations are reported in Tables 14 and 15 respectively.

On WebNLG 2017 dataset, three metrics achieve similar scores for semantic adequacy, with no results significantly larger than the others (computed
<table><caption>Table 16: Krippendorff's alpha for ordinal metrics on WebNLG 2017 human annotations</caption><thead><tr><td>Metric</td><td>Krippendorff's Alpha</td></tr></thead><tbody><tr><td>Fluency</td><td>0.4201</td></tr><tr><td>T. Struct.</td><td>0.3231</td></tr><tr><td>Sem. Adeq.</td><td>0.5314</td></tr></tbody></table>

<table><caption>Table 17: Krippendorff's alpha for ordinal metrics on WebNLG 2020 human annotations</caption><thead><tr><td>Metric</td><td>Krippendorff's Alpha</td></tr></thead><tbody><tr><td>Correct.</td><td>0.2769</td></tr><tr><td>D. Cover.</td><td>0.2632</td></tr><tr><td>Fluency</td><td>0.2580</td></tr><tr><td>Relev.</td><td>0.1860</td></tr><tr><td>T. Struct.</td><td>0.2234</td></tr></tbody></table>

using confidence intervals by bootstrapping technique in Appendix A.2). Using (Evans, 1996) correlation guidelines, where a value between 0.6 to 0.79 is "strong correlation" and 0.8 to 1.0 is "very strong", we have "strong" correlation for the type of sentence correlation to the annotation. However, on WebNLG 2020 dataset, all metrics demonstrate a "moderate" level of correlation, given that the scores hover between 0.3 and 0.49.

We note that the two sentence correlation interpretations presented above are more sensitive to noise in the human annotation, as we consider each individual sentence score, not aggregates. This is evident also from the lower Human-Human correlation we observe on sentences, in comparison with the high correlation for the system level correlation. In Tables 16 and 17, we present the inter-annotator agreement for WebNLG 2017 and 2020 using Krippendorff's alpha (Krippendorff, 2018). It can be observed that both datasets exhibit generally low consistency among annotators, with the 2020 dataset showing less agreement than its 2017 version. However, on WebNLG 2017 we observe a higher agreement on semantic adequacy, for which we also observed a high correlation with our metric at sentence level.

We believe that these results show the need for better human annotation guidelines for this task. In particular, on WebNLG2020, the annotators were asked to give a score from 0 to 100 to a sentence for a given dimension such as correctness. Such a fine-grained decision is very difficult to take, hence the low agreement score. We note that mapping these scores to scores from 1 to 10 or 1 to 5 does

not improve agreement. In the 2017 challenge, the annotators gave scores from 1 to 3, but we believe that those scores were not sufficiently described such that the annotators could choose them accordingly. We recommend that future annotations use a Likert scale annotation, but with clearer guidelines and examples for each score.

## A.6 Qualitative Analysis for examples with different number of triples

We conducted a qualitative analysis on examples with different number of triples, to observe where the proposed metric shows improvements during decoding. We present the number of (graph, text) pairs as categorized by the number of triples within the input graph for the test set, and results for various datasets segmented by the number of triples are reported below. SimpleQuestion was not considered because it has only one triple per input graph.

For the WebNLG 2017 dataset, we can observe that for the small G2T model, T5S, the triples of size 4, 5 and 7 are more challenging, as is shown in Table 19. The integration of FactSpotter enhances the factuality of the results (as evident in F-T5S). We also observed a distinct improvement across various numbers of triples in the WebNLG 2020 dataset, especially when handling sentences comprising multiple triples.

As for the DART dataset in Table 23, inputs ranging from 1 to 5 triples witness improved results with FactSpotter's addition. Remarkably high FactSpotter scores are observed for inputs containing 6 or 7 triples. It's worth noting that of the inputs with 6 triples, 569 out of 598 are sourced from the E2E split, and for those with 7 triples, 320 out of 342 hail from the same E2E split. Hence, the E2E split of DART seems to be less challenging.

The results of GrailQA dataset are illustrated in Tables 25, 29, and 27. Regarding the GrailQA zero-shot split, single-triple verbalization consistently achieves superior scores with FactSpotter. Beginning with two triples, the incorporation of FactSpotter offers a discernible boost in model performance. In the compositional split, mirroring the trend observed in the zero-shot scenario, FactSpotter scores remain high for single-triple inputs. Furthermore, there is a marked improvement for inputs with two or more triples upon the addition of FactSpotter. In the IID split, while the baseline scores are impressively high, enhancements is noticeable for small model like T5S.
<table><caption>Table 18: The number and the percentage of (graph, sentence) pairs for the test set of WebNLG 2017 Const</caption><thead><tr><th>Triple Num</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th></tr></thead><tbody><tr><th>Pair Num</th><td>1540</td><td>821</td><td>419</td><td>241</td><td>185</td><td>0</td><td>6</td></tr><tr><th>Percentage</th><td>47.95</td><td>25.56</td><td>13.04</td><td>7.50</td><td>5.76</td><td>0</td><td>0.19</td></tr></tbody></table>

<table><caption>Table 19: FactSpotter by the number of triples for the test set of WebNLG 2017 Const</caption><thead><tr><th>Triple Num</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>7</th></tr></thead><tbody><tr><th>T5S</th><td>99.87</td><td>99.45</td><td>99.13</td><td>97.62</td><td>98.09</td><td>97.53</td></tr><tr><th>F-T5S</th><td>99.75</td><td>99.65</td><td>99.22</td><td>98.74</td><td>99.20</td><td>99.93</td></tr><tr><th>T5B</th><td>99.65</td><td>99.69</td><td>99.35</td><td>99.15</td><td>99.52</td><td>99.93</td></tr><tr><th>F-T5B</th><td>99.81</td><td>99.70</td><td>99.69</td><td>99.85</td><td>99.53</td><td>99.93</td></tr><tr><th>JGT-T5</th><td>99.69</td><td>99.73</td><td>98.52</td><td>98.63</td><td>99.30</td><td>99.93</td></tr><tr><th>FGT-T5</th><td>99.40</td><td>99.89</td><td>99.50</td><td>99.89</td><td>99.60</td><td>99.93</td></tr></tbody></table>

<table><caption>Table 20: The number and the percentage of (graph, sentence) pairs for the test set of WebNLG 2020</caption><thead><tr><th>Triple Num</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th></tr></thead><tbody><tr><th>Pair Num</th><td>369</td><td>698</td><td>1050</td><td>1220</td><td>1065</td><td>684</td><td>553</td></tr><tr><th>Percentage</th><td>9.09</td><td>17.19</td><td>25.86</td><td>30.04</td><td>26.21</td><td>16.84</td><td>13.62</td></tr></tbody></table>

<table><caption>Table 21: FactSpotter by the triple number for the test set of WebNLG 2020</caption><thead><tr><th>Triple Num</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th></tr></thead><tbody><tr><th>T5S</th><td>93.71</td><td>93.91</td><td>92.24</td><td>90.71</td><td>89.95</td><td>88.68</td><td>86.12</td></tr><tr><th>F-T5S</th><td>95.05</td><td>95.13</td><td>95.58</td><td>94.07</td><td>92.21</td><td>91.01</td><td>90.54</td></tr><tr><th>T5B</th><td>95.21</td><td>96.33</td><td>94.51</td><td>93.00</td><td>92.29</td><td>92.30</td><td>93.36</td></tr><tr><th>F-T5B</th><td>96.27</td><td>96.74</td><td>95.69</td><td>94.41</td><td>94.72</td><td>94.89</td><td>94.09</td></tr><tr><th>JGT-T5</th><td>94.74</td><td>94.71</td><td>92.71</td><td>92.16</td><td>88.16</td><td>89.10</td><td>88.46</td></tr><tr><th>FGT-T5</th><td>94.26</td><td>95.85</td><td>93.85</td><td>93.93</td><td>90.90</td><td>91.43</td><td>92.48</td></tr></tbody></table>

<table><caption>Table 22: The number and the percentage of (graph, sentence) pairs for DART test set</caption><thead><tr><th>Triple Num</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th></tr></thead><tbody><tr><th>Pair Num</th><td>848</td><td>797</td><td>821</td><td>869</td><td>822</td><td>598</td><td>342</td></tr><tr><th>Percentage</th><td>16.64</td><td>15.64</td><td>16.11</td><td>17.05</td><td>16.13</td><td>11.73</td><td>6.71</td></tr></tbody></table>

<table><caption>Table 23: FactSpotter by triple number for DART</caption><thead><tr><th>Triple Num</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th></tr></thead><tbody><tr><th>T5S</th><td>98.47</td><td>93.55</td><td>93.35</td><td>92.45</td><td>94.01</td><td>99.57</td><td>99.82</td></tr><tr><th>F-T5S</th><td>99.71</td><td>97.21</td><td>96.41</td><td>94.82</td><td>96.68</td><td>99.88</td><td>99.82</td></tr><tr><th>T5B</th><td>98.08</td><td>95.53</td><td>94.54</td><td>94.44</td><td>96.03</td><td>99.88</td><td>99.99</td></tr><tr><th>F-T5B</th><td>99.54</td><td>96.81</td><td>96.32</td><td>96.16</td><td>97.19</td><td>99.74</td><td>99.87</td></tr><tr><th>JGT-T5</th><td>98.49</td><td>94.36</td><td>93.61</td><td>93.44</td><td>94.33</td><td>99.82</td><td>99.80</td></tr><tr><th>FGT-T5</th><td>99.51</td><td>96.43</td><td>95.83</td><td>94.78</td><td>95.82</td><td>99.77</td><td>99.89</td></tr></tbody></table>

## A.7 Further analysis on the difficulty of graph-to-text generation

We consider the intricacy in verbalizing a given triple is predominantly influenced by:

1. The distance between entity names in the knowledge graph and those in natural text.

2. The difference between KG predicates and their equivalent natural language phrases.

<table><caption>Table 24: The number and the percentage of (graph, sentence) pairs for GrailQA zero-shot split</caption><thead><tr><th>Triple Num</th><th>1</th><th>2</th><th>3</th><th>4</th></tr></thead><tbody><tr><th>Pair Num</th><td>1231</td><td>522</td><td>55</td><td>20</td></tr><tr><th>Percentage</th><td>67.34</td><td>28.55</td><td>3.00</td><td>1.09</td></tr></tbody></table>

<table><caption>Table 25: FactSpotter by the number of triples for GrailQA Zero-shot split</caption><thead><tr><th>Triple Num</th><th>1</th><th>2</th><th>3</th><th>4</th></tr></thead><tbody><tr><th>T5S</th><td>96.41</td><td>89.63</td><td>92.71</td><td>93.62</td></tr><tr><th>F-T5S</th><td>97.08</td><td>91.30</td><td>94.52</td><td>99.39</td></tr><tr><th>T5B</th><td>96.49</td><td>93.03</td><td>92.84</td><td>94.98</td></tr><tr><th>F-T5B</th><td>97.48</td><td>93.31</td><td>94.28</td><td>99.68</td></tr><tr><th>JGT-T5</th><td>96.96</td><td>90.77</td><td>93.50</td><td>91.28</td></tr><tr><th>FGT-T5</th><td>98.02</td><td>91.63</td><td>95.95</td><td>93.81</td></tr></tbody></table>

<table><caption>Table 26: The number and the percentage of (graph, sentence) pairs for GrailQA Compositional split</caption><thead><tr><th>Triple Num</th><th>1</th><th>2</th><th>3</th><th>4</th></tr></thead><tbody><tr><th>Pair Num</th><td>388</td><td>292</td><td>39</td><td>38</td></tr><tr><th>Percentage</th><td>51.25</td><td>38.57</td><td>5.15</td><td>5.01</td></tr></tbody></table>

<table><caption>Table 27: FactSpotter by the number of triples for GrailQA Compositional split</caption><thead><tr><th>Triple Num</th><th>1</th><th>2</th><th>3</th><th>4</th></tr></thead><tbody><tr><th>T5S</th><td>98.47</td><td>94.89</td><td>95.53</td><td>88.15</td></tr><tr><th>F-T5S</th><td>99.14</td><td>95.77</td><td>98.12</td><td>94.69</td></tr><tr><th>T5B</th><td>98.64</td><td>96.23</td><td>77.78</td><td>93.27</td></tr><tr><th>F-T5B</th><td>99.25</td><td>96.35</td><td>93.80</td><td>94.36</td></tr><tr><th>JGT-T5</th><td>98.84</td><td>95.59</td><td>89.73</td><td>91.02</td></tr><tr><th>FGT-T5</th><td>98.92</td><td>96.62</td><td>98.79</td><td>94.85</td></tr></tbody></table>

<table><caption>Table 28: The number and the percentage of (graph, sentence) pairs for GrailQA IID split</caption><thead><tr><th>Triple Num</th><th>1</th><th>2</th><th>3</th><th>4</th></tr></thead><tbody><tr><th>Pair Num</th><td>257</td><td>465</td><td>63</td><td>12</td></tr><tr><th>Percentage</th><td>32.65</td><td>58.34</td><td>7.90</td><td>1.50</td></tr></tbody></table>

<table><caption>Table 29: FactSpotter by the number of triples for GrailQA IID split</caption><thead><tr><th>Triple Num</th><th>1</th><th>2</th><th>3</th><th>4</th></tr></thead><tbody><tr><th>T5S</th><td>97.75</td><td>97.98</td><td>98.82</td><td>96.96</td></tr><tr><th>F-T5S</th><td>97.96</td><td>98.73</td><td>98.83</td><td>99.20</td></tr><tr><th>T5B</th><td>99.55</td><td>99.31</td><td>99.38</td><td>99.77</td></tr><tr><th>F-T5B</th><td>99.55</td><td>99.47</td><td>99.38</td><td>99.77</td></tr><tr><th>JGT-T5</th><td>98.09</td><td>99.12</td><td>98.31</td><td>99.77</td></tr><tr><th>FGT-T5</th><td>98.72</td><td>99.35</td><td>99.31</td><td>99.77</td></tr></tbody></table>
**Entity Generation** Regarding the difficulty of generating correct entities, we have the following statistics on the WebNLG 2017 dataset (v2.1):

* 87% of input graph entities appear identical in its corresponding ground-truth texts.

* 7% of differences arise from special characters, e.g., "Motherwell F.C." in the input graph becomes "Motherwell FC" in the ground-truth, both being correct.

* 2% are due to variations in date or number formats, as observed with "2006-12-31" in the KG interpreted to be "December 31, 2006" in the ground-truth text.

* 1% emerge from linguistic differences between the source language and English, as with "Atatürk" in KG being written as "Ataturk" in reference text.

* Other variances often regard alternative entity designations, such as "United States" in the KG being abbreviated as "US" in specific ground-truth texts.

For the GrailQA dataset, rooted in Freebase, 99% of input graphs retain consistent entity names in their ground-truth sentences. Accurate entity verbalization is relatively straightforward for this dataset, but we observe hallucinations in the texts generated by baseline models.

Considering that most differences between KG and natural language entity names result from formatting nuances, we can easily address its consistency challenge using constrained beam search to ensure all entity names appear in the generated text. For beam sizes over 10, the text generation becomes accurate without compromising other metrics. However, for beam sizes under 10, such constraints tend to impede the fluency.

**Predicate Generation** In the WebNLG 2017 dataset, 49% of predicates appear the same as in the KB in their corresponding ground-truth sentences. Using SBert, we computed an average similarity of 84% between each predicate and its nearest n-grams in the ground-truth.

For the GrailQA dataset, only 27% of predicates align perfectly with their KG representations in the ground-truth. The average similarity stands at 64%, highlighting a larger difference between FreeBase predicates and their natural language phrases.

**Summary of Difficulty** Datasets like WebNLG (from DBPedia) and GrailQA (from FreeBase) present challenges on different fronts. The difficulty from datasets based on FreeBase is the distance between knowledge graph and natural language is much higher. However, WebNLG and DART datasets have more complex input subgraphs, which has more number of triples, while GrailQA only has input subgraphs with up to 4 triples.

We consider that promoting accurate predicate generation is more challenging problem than promoting the generation of correct entities, because predicates are much more often rephrased in sentences, which is harder to evaluate. Inspired by the efficacy of Constrained Beam Search in ensuring accurate entity generation, we designed *FactSpotter* to enhance the accurate production of rephrased facts, especially rephrased predicates to be correctly generated.