# Semantic Evaluation of Multilingual Knowledge Graph-to-Text Generation via NLI Fine-Tuning: Precision, Recall and F1 scores

William Soto Martinez, Yannick Parmentier, Claire Gardent

► To cite this version:

William Soto Martinez, Yannick Parmentier, Claire Gardent. Semantic Evaluation of Multilingual Knowledge Graph-to-Text Generation via NLI Fine-Tuning: Precision, Recall and F1 scores. The 63rd Annual Meeting of the Association for Computational Linguistics, Jul 2025, Vienna, Austria. hal-05138142

HAL Id: hal-05138142

https://hal.science/hal-05138142v1

Submitted on 1 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Semantic Evaluation of Multilingual Knowledge Graph-to-Text Generation
via NLI Fine-Tuning: Precision, Recall and F1 scores

William Soto Martinez

Université de Lorraine / LORIA

william-eduardo.soto-martinez@loria.fr

Yannick Parmentier

Université de Lorraine / LORIA

yannick.parmentier@loria.fr

Claire Gardent

CNRS/LORIA and Université de Lorraine

claire.gardent@loria.fr

## Abstract

Performance in the Knowledge Graph-to-Text generation has improved over time, particularly in English. However, models are still prone to mistakes like Additions and Omissions. Furthermore, few languages are taken into account since both train and test data are not readily available. In this paper, we hope to facilitate the development and improvement of multilingual KG-to-Text models by providing a multilingual evaluation framework that is reference-less and permits estimating how much a KG-to-Text Model under- (omission) or over- (addition) generates. We focus on two high (English, Russian) and five low (Breton, Irish, Maltese, Welsh, Xhosa) resource languages and show that our metric has fair to moderate correlation with reference-based metrics, positioning it as a consistent alternative when no references are available. We also show that our metric outperforms prior reference-less metrics in correlation with existing human judgments. Additional human evaluation shows moderate to strong correlation with human annotators in assessing precision and recall at a higher granularity level than shown in previous studies. Since our metric provides scores for precision and recall, it helps better assess the level of over- or under-generation of multilingual KG-to-Text models. We make our data, code and models available<fnref n="1" />.

## 1 Introduction

Figure 1 shows an example of a Knowledge Graph and its verbalization. In a Knowledge Graph (KG), each edge represents a fact as a (subject, predicate, object) triple. To make these graphs more accessible, KG-to-Text generation models have been proposed whose function is to convert KGs into natural language. A key constraint on this task is that generation should be semantically faithful,

<figure><img src="image_2.png" /><figcaption>Figure 1: Example of a Knowledge Graph from the DBpedia Knowledge Base together with a possible correct verbalization.</figcaption></figure>

meaning that the generated text should express all and only the content represented by the input KG.

While KG-to-Text generation models have steadily improved over the years both in terms of performance and of range of target languages they can handle (Gardent et al., 2017; Castro Ferreira et al., 2020; Cripwell et al., 2023), recent results indicate that semantic faithfulness is still an issue since the generated texts can either contain information not present in the input (Additions) or, conversely, fail to express all the information present in the input (Omissions). These issues are particularly prevalent when generating into under-resourced languages (Cripwell et al., 2023) or out-of-domain topics (Nikiforovskaya and Gardent, 2024).

In this paper, we provide a novel framework for the evaluation of KG-to-Text Models which, we hope, will help support the development of multilingual, semantically faithful KG-to-Text models. We make the following contributions:

1) A new reference-less multilingual metric that quantifies how much a model under- (omissions) or over- (additions) generates. This metric provides three scores: precision, recall, and F1. Intuitively, the graph acts as a reference. Hence, precision is the ratio between correct information in the text

<footnote n="1">¹https://gitlab.inria.fr/wsotomar/semantic-evaluation-of-multilingual-data-to-text-generation-via-nli-fine-tuning</footnote>
<table><caption>Table 1: Example KG graph and some possible lexicalizations of this graph. The lexicalizations have different precision and recall scores as well as an explanation of the errors that cause the scores (O:Omission, A:Addition, A triple that is not fully expressed by the text counts as omitted and vice versa for additions).</caption><thead><tr><th colspan="4">Graph</th></tr><tr><th colspan="4">Alan Bean | birthDate | 1932-03-15</th></tr><tr><th colspan="4">Alan Bean | almaMater | UT Austin, B.S. 1955</th></tr><tr><th colspan="4">Alan Bean | birthPlace | Wheeler, Texas</th></tr><tr><th>Texts</th><th>Precision</th><th>Recall</th><th>Errors</th></tr></thead><tbody><tr><td>Alan Bean was born on March 15, 1932.</td><td>1/1</td><td>1/3</td><td>20</td></tr><tr><td>Alan Bean was born in Wheeler, Texas and was in the Apollo 12 mission.</td><td>1/2</td><td>1/3</td><td>1A, 2O</td></tr><tr><td>Alan Bean was born on March 15, 1932 in Wheeler, Texas.<br>He received a Bachelor of Science degree at the<br>University of Texas at Austin in 1955.</td><td>3/3</td><td>3/3</td><td>None</td></tr></tbody></table>

and total information in the text (how much of the generated text is correct?) while recall is the ratio between correct information in the text and information in the input graph (how much of the input graph does the text convey?). F1 is their harmonic mean (See Table 1 for some illustrating examples).

2) A methodology for creating the training data necessary to train our metric.

3) Testing on both high (English, Russian) and low (Breton, Irish, Maltese, Welsh, Xhosa) resource languages, we compute correlation with both existing reference-based metrics and human judgments. We show that correlation with reference-based metrics is fair to moderate, which indicates that our metric, although reference-less, can be used to a certain extent in place of reference-based metrics; particularly when references are not available. When comparing with human judgments, we show that correlation with our metric outperforms the correlation obtained on the same data by other existing reference-less metrics developed for English KG-to-Text like Data-QuestEval (Rebuffel et al., 2021) and FactSpotter (Zhang et al., 2023).

## 2 Related Work

**Zero-shot NLI Classification for Semantic Accuracy.** Dušek and Kasner (2020) proposed to evaluate the semantic accuracy of English KG-to-Text generation by leveraging the zero-shot abilities of the English-based RoBERTa-Large-NLI model (Liu et al., 2019)<fnref n="2" />. In their work, they try two approaches: one to search for Omissions and one to search for Additions. In the first one, they use the entire generated text as a premise and iterate over every individual fact from the input graph

as a hypothesis, marking facts not classified as entailments as Omissions. In the second approach, they use the entire input graph as a premise and the generated text as a hypothesis, marking the presence of Additions when the text is not classified as entailment. This approach is tested exclusively in English and, while it provides high granularity when measuring Omissions, it is less specific when measuring Additions.

**Fine-tuning on Synthetic Data for Factual Faithfulness.** Zhang et al. (2023) went a step further and fine-tuned an English model, first an Electra-Base-Discriminator (Clark et al., 2020)<fnref n="3" /> then a DeBERTa-V3-Base (He et al., 2021)<fnref n="4" />, on synthetic data to detect whether a given fact is present in a generated text (akin to Dušek and Kasner's Omission check). Their training data consist of real positive (Text, Fact) pairs and synthetic negative (Text, Fact) pairs made of 90% type I errors (where they perturbed the fact by changing its subject, predicate, and/or object) and 10% type II errors (where they perturbed the text by removing one or both entities from the fact and/or the n-grams most similar to the predicate). Like FactSpotter, this approach was tested exclusively in English and, while it has a high granularity when measuring Omissions, it does not directly address Additions.

**Beyond KG-to-Text:** Bidirectional entailment has been tried as a way of evaluating summarization. Kane et al. (2020) and Zhang and Perez-Beltrachini (2024) proposed reference-based approaches, while Chen and Eger (2023) proposed a reference-less method.

<footnote n="2">²https://huggingface.co/FacebookAI/roberta-large-mnli</footnote>

<footnote n="3">³https://huggingface.co/google/electra-base-discriminator</footnote>

<footnote n="4">⁴https://huggingface.co/microsoft/deberta-v3-base</footnote>
## 3 Method

To learn our metric, we fine-tune an existing multi-lingual Natural Language Inference (NLI) model by adjusting its classification head to perform regression instead. We then train on data created to capture different combinations of precision and recall using Binary Cross Entropy (BCE) loss.

Given a premise and a hypothesis, NLI models predict if the hypothesis is entailed, neutral, or contradicted by the premise. For precision, we check if the text is entailed by the graph (how much of the text can be inferred from the graph). For recall we check if the graph is entailed by the text (how much of the graph content can be inferred from the text).

We are not interested in the 3 classes from the NLI head, only in the strength of the entailment between premise and hypothesis. We fine-tune the NLI classifier as a regression model by focusing only on the entailment weights from the classification head instead of the three existing output classes. We train simultaneously for precision and recall by swapping the graph and text order and targeting the respective score.

The F1 score is computed as usual (Equation 1) by taking the harmonic mean of Precision (p) and Recall (r). This score functions as a high-level proxy for semantic faithfulness: the higher the F1 score, the higher the semantic similarity between the Graph and the Text.

$$F_1 = 2 \frac{p \cdot r}{p + r} \qquad (1)$$

### 3.1 Training Data Creation

We aim to generate a training dataset of (graph, text, precision, recall) quadruples with a balanced and diverse distribution of precision and recall.

First we collect the set G of WebNLG graph/text pairs from the English WebNLG V3.0 dataset (Castro Ferreira et al., 2020)<fnref n="5" />. This dataset is semantically aligned (graph and text match in content), so we assign each $(g, t) \in G$ precision and recall scores of 1.

To increase precision and recall diversity we derive non aligned $(g', t)$ pairs from $(g, t) \in G$ by pairing the text $t$ with graphs $g'$ which $i$) are subgraphs or super graphs of $g$ or ii) modify $g$ either by adding to it triples from non overlapping graphs or by modifying a triple contained in $g$. We then compute precision and recall for each new $(g', t)$ pair based on the number of added, removed or modified triples.

Once we have a balanced English dataset, we extend it to other languages by machine translating the English texts. We use NLLB-200-3.3B (Team et al., 2022)<fnref n="6" /> to translate into five languages: Irish, Maltese, Russian, Welsh, and Xhosa. To reduce the noise introduced by machine translation, we filter these translations following two criteria: Language Identification score via GlotLID (Kargaran et al., 2023)<fnref n="7" /> and LID218e (Team et al., 2022)<fnref n="8" />, and Semantic Similarity score via LaBSE (Feng et al., 2022)<fnref n="9" />.

Figure 2 shows the distribution of precision and recall scores in 1.77 million (graph, text, precision, recall) quadruples evenly distributed across six languages. Appendix A provides more details.

<figure><img src="image_3.png" /><figcaption>Figure 2: Number of samples by precision and recall scores in the training dataset.</figcaption></figure>

<footnote n="5">⁵https://gitlab.com/shimorina/webnlg-dataset /-/tree/master/release_v3.0</footnote>

<footnote n="6">⁶https://huggingface.co/facebook/nllb-200-3.3 B</footnote>

<footnote n="7">⁷https://huggingface.co/cis-lmu/glotlid</footnote>

<footnote n="8">⁸https://huggingface.co/facebook/fasttext-lan guage-identification</footnote>

<footnote n="9">⁹https://huggingface.co/sentence-transformers/ LaBSE</footnote>
## 3.2 Models

We use the following three models as baselines.

**Data-QuestEval(DQE).** A reference-less model by Rebuffel et al. (2021) which relies on question-generation and question-answering to assess semantic faithfulness. The main limitations of this baseline are that it was only fine-tuned for English, the long processing time of the text generation, and the risk of generating questions and answers unrelated to the actual input.

**FactSpotter(FS).** The latest model by Zhang et al. (2023)<fnref n="10" />. Compared to our approach, the main limitations of this baseline are that it was only fine-tuned for English and that it produces a single, recall oriented score.

**NLI Base (NB).** This baseline follows the exact same process as (Dušek and Kasner, 2020). We only change the off-the-shelf NLI model they used for a Multilingual one (Laurer et al., 2022)¹¹ instead of an English one. The main limitations of this baseline are that it is not familiar with the KG format and that it has not seen all the target languages we will test.

To learn our metric, we fine-tune mDeBERTa-v3-base-xnli-multilingual-nli-2mil7 (NB) on the dataset from Section 3.1. We fine-tune it as a regression model by targeting the entailment weights of the classification head and training simultaneously for precision and recall. For the precision score, we use the graph as premise and the text as hypothesis; for the recall score, we use the text as premise and the graph as the hypothesis. We fine-tune and compare three versions:

**Multilingual Full Fine-Tuning (MultiFF):** Full fine-tuning of the NLI Base model on all languages together.

**Multilingual LoRA (MultiLR):** LoRA on top of the NLI-Base model on all languages together.

**Monolingual LoRA (MonoLR):** Lora on top of the NLI-Base model for each language individually.

Appendix B provides details on the hyper parameters used to train our models.

# 4 Evaluation

We evaluate our metrics using correlation with human judgments (6 languages) and with automatic metrics (7 languages). We also report results on KG/text retrieval accuracy (7 languages). Table 2 summarises the test sets used.

## 4.1 Correlation with Automatic Metrics

Here, we use the 7L-Auto dataset, which consists of all graphs from the WebNLG test data<fnref n="12" /> and all the texts generated from these graphs by participant systems of the WebNLG 2017, 2020, and 2023 Shared Tasks, as well as the different models trained by Meyer and Buys (2024). The models used to generate the texts include grammar-based and template-based approaches, statistical MT models, neural models trained from scratch, and fine-tuned pretrained models, covering a broad spectrum of errors and quality levels. Texts are generated in English, Russian, Breton, Irish, Maltese, Welsh, and Xhosa.

We compute the Spearman's Correlation (ρ) of the baselines and of our models with 5 reference-based metrics: BLEU (Papineni et al., 2002), ChrF++ (Popović, 2017), TER (Olive, 2005), BERTScore (Zhang* et al., 2020), and SBERT similarity (Reimers and Gurevych, 2019). For TER we report the inverse score (¬TER = 1 – TER) for easier display.

<table><caption>Table 2: Datasets used for Correlation Studies. When a relevant annotation is adjacent to Precision (p) or Recall (r) that is indicated, otherwise the annotation is consider adjacent to the more general F1 score.</caption><thead><tr><td>Dataset</td><td># Graphs</td><td># Texts</td><td>Languages</td><td>Relevant Annotations</td></tr></thead><tbody><tr><td>7L-Auto</td><td>4461</td><td>143 838</td><td>br, cy, en, ga, mt, ru, xh</td><td>BLEU, ChrF++, TER, BERTScore, SBERT</td></tr><tr><td>4L-RP-Human</td><td>181</td><td>200</td><td>cy, en, mt, ru</td><td>Precision(p) and Recall(r)</td></tr><tr><td>2017</td><td>223</td><td>2 230</td><td>en</td><td>Semantics</td></tr><tr><td>2020</td><td>288</td><td>3 905</td><td>en, ru</td><td>Relevance(p), Correctness(p), Data Coverage(r)</td></tr><tr><td>2023</td><td>200</td><td>1 700</td><td>cy, ga, ru, mt</td><td>Omissions(r), Additions(r)</td></tr></tbody></table>

<footnote n="12">¹²Specifically, we use the graphs from the WebNLG 2017 test set (1,862 graphs), from the WebNLG 2020 test set for English (1,779), from the WebNLG 2020 test set for Russian that are not present in the English test set (732) and from the Xhosa data sets that are not in any of the other datasets (88).</footnote>

<footnote n="10">¹⁰Inria-CEDAR/FactSpotter-DeBERTaV3-Base
¹¹https://huggingface.co/MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7</footnote>
A good correlation would indicate that in the absence of ground truth, our metrics can be used as a proxy for these reference based metrics.

## 4.2 Correlation with Human Judgments

We use human judgments collected by the WebNLG campaigns and a subset we created to have human annotations specifically targeting precision and recall.

**WebNLG 2017.** The human annotations for this challenge (Shimorina et al., 2018) consist of 223 graphs lexicalized in English by 9 different NLG systems, plus the human-written references. The generations were scored on a 3-point Likert scale across 3 criteria: Fluency, Grammar, and Semantics. For this study, we focus on the Semantics annotation:

* Semantics: *Does the text correctly represent the meaning in the data?*

Since the Semantics annotation does not specify the type of error (Additions, Omission, etc.) we compute correlation between the WebNLG 2020 Semantics score and our F1 score.

**WebNLG 2020.** The human annotations for this challenge (Castro Ferreira et al., 2020) consist of 178 graphs lexicalized in English by 16 different NLG systems and 110 graphs lexicalized in Russian by seven different NLG systems; additionally, both include their human written references. The generations were scored on a 0 to 100 scale across five criteria: Text Structure, Fluency, Relevance, Correctness, and Data Coverage. For this study, we focus on the last three:

* Relevance: *Does the text describe only such predicates (with related subjects and objects), which are found in the data?*

* Correctness: *When describing predicates which are found in the data, does the text mention correct the objects and adequately introduces the subject for this specific predicate?*

* Data Coverage: *Does the text include descriptions of all predicates presented in the data?*

When computing correlation, we compare precision with the product of the Relevance and Correctness, recall with Data Coverage, and F1 with the harmonic means of both.

**WebNLG 2023.** The human annotations for this challenge (Cripwell et al., 2023) consist of 100 graphs lexicalized in Irish by 4 NLG systems, Maltese by 3 NLG systems, Welsh by 3 NLG systems, and other 100 graphs lexicalized in Russian by 3 NLG systems. Additionally, all of them included their human-written references. The generations were scored across 4 criteria: Fluency, Absence of Unnecessary Repetition, Absence of Additions, and Absence of Omissions. The first is on a 5-point Likert scale; the other 3 have binary Yes/No labels. For this study, we focus on Absence of Additions and Absence of Omissions:

* Absence of Additions: *Looking at the Text, is all of its content expressed in the Data expression? (Allow duplication of content.)*

* Absence of Omissions: *Looking at each element of the Data expression in turn, does the Text express all the information in all elements in full (allow synonyms and aggregation)?*

We compute correlation between precision and Absence of Additions, recall and Absence of Omissions, and F1 with their harmonic mean.

**4L-RP-Human.** While WebNLG's existing human judgments can, to a certain extent, be used as proxies for Precision, Recall, and F1, none of them were collected to measure these values specifically. To address this, we created a new dataset of human judgments called 4L-RPHuman, with KG/text pairs extracted from the 7L-Auto dataset. It contains 50 KG/text pairs per language for four languages (English, Maltese, Russian, and Welsh) with a balanced distribution of Precision and Recall scores by our best-performing model. We then obtain human annotations for Precision and Recall of this subset to test how our model correlates with human judgments that specifically target these properties. The human annotators were provided with a text and a graph in table format and were asked to answer, using a scale of 1 to 5 (None, Few, Half, Most, All), the following questions:

* Precision: *How many Triples from the text can you find in the Table?*

* Recall: *How many Triples from the table can you find in the Text?*

The annotators were native speakers of the target language who were proficient in English hired via
Prolific<fnref n="13" /> and paid 10£/h. Inter-annotator agreement was measured via Fleiss' Kappa (Fleiss, 1971). Appendix G provides more details.

## 4.3 Retrieval Accuracy

We also evaluate how well the scoring of various models discerns between good and bad pairings using a retrieval reformulation of the KG-to-text task: Given the embedding of a graph, how well can the model identify the most similar text in a corpus and vice versa given a text how well can it identify the corresponding graph?

Given a subset of 100 KG/text pairs randomly selected from the WebNLG dataset for each target language, we compute the F1 score with our model

and the score produced by each of the baselines for each of the 10K possible graph/text combinations. We then compute Retrieval at 1 (A@1) i.e., the proportion of cases where the highest score is assigned to the correct graph-text pair. We limit the size of this subset given the computational demands of scoring all possible combinations of graphs and texts with our cross-encoder approach.

# 5 Results

## 5.1 Correlation with Automatic Metrics

Figure 3 shows the Spearman's Correlation (ρ) between the various reference-less metric we evaluate and reference-based automatic metrics on the 7L-Auto dataset (Breton, English, Irish, Maltese, Russian, Welsh and Xhosa WebNLG generations).

<figure><img src="image_4.png" /><figcaption>Figure 3: Spearman's Correlation (ρ) between reference-less and reference-based metrics on the 7L-Auto dataset. Only results with a p-value under 0.05 are reported. NB: Since there is no Breton training data, the MonoLR score for Breton is computed with its closest language (Welsh). Xhosa results are provided in Appendix C.</figcaption></figure>

<footnote n="13">¹³https://www.prolific.com/</footnote>
**Fine-tuning matters.** Models that have been fine-tuned for the task (including the FS baseline) show positive correlation across metrics and languages, while the NB baseline has either negative or non significant correlations. These results highlight the limitations of using off-the-shelf models as proposed in (Dušek and Kasner, 2020) and underscore the importance of task specific fine-tuning.

**Strong performance in English.** While trained on multilingual data, our models almost match the performance of metrics trained on English only (DQE, FS, NB). Interestingly, the gap is smallest for semantic based metrics (SBERT, BERTScore), suggesting that our metrics are good at capturing paraphrases.

**Good performance in other languages.** Our fine-tuned models, especially the small Monolingual LoRA version, outperform all three baselines in all the other languages. These results demonstrate the effectiveness of our approach despite fine-tuning on synthetic, non-gold data.

## 5.2 Correlation with Human Judgments

### 5.2.1 Evaluation on Human Judgments for Precision and Recall.

Table 3 reports correlation results when comparing the precision, recall and F1 scores predicted by our models with corresponding human judgments (4L-RP-Human dataset) and the inter-annotator agreement (Fleiss κ). They show a strong correlation for all three metrics in English, Russian and Welsh and a moderate one for Maltese showing the effectiveness of our approach to capture omissions (recall), addition (precision) and semantic faithfulness (F₁).

We provide examples illustrating good, medium and bad output from our best model (MonoLR) for each target language in Table 13, Table 14, Table 15, and Table 16 (Appendix F).

### 5.2.2 Evaluation on Human Judgments from the WebNLG Shared Tasks.

Figure 4 shows the Root Mean Squared Error (RMSE) and Spearman's correlation ($\rho$) of the F1 score of different automatic metrics against the WebNLG 2017, 2020 and 2023 human annotations.

<table><caption>Table 3: Fleiss' κ of Precision and Recall Human Judgments as well as the Spearman's Correlation ($\rho$) of their average compared to our MonoLR model on the 4L-RP-Human subset</caption><thead><tr><th rowspan="2">Language</th><th rowspan="2">Annotators</th><th colspan="2">Precision</th><th colspan="2">Recall</th><th rowspan="2">F1<br>ρ</th></tr><tr><th>Fleiss κ</th><th>ρ</th><th>Fleiss κ</th><th>ρ</th></tr></thead><tbody><tr><td>English</td><td>4</td><td>0.47</td><td>0.68</td><td>0.47</td><td>0.63</td><td>0.70</td></tr><tr><td>Maltese</td><td>3</td><td>0.29</td><td>0.38</td><td>0.49</td><td>0.30</td><td>0.47</td></tr><tr><td>Russian</td><td>2</td><td>0.32</td><td>0.63</td><td>0.39</td><td>0.52</td><td>0.67</td></tr><tr><td>Welsh</td><td>4</td><td>0.37</td><td>0.60</td><td>0.50</td><td>0.81</td><td>0.70</td></tr></tbody></table>

<figure><img src="image_5.png" /><figcaption>Figure 4: Root Mean Squared Error (RMSE) and Spearman's correlation ($\rho$) of the F1 score from different automatic metrics against the closest approximate human annotations from WebNLG 2017, 2020 and 2023 (Each year has different annotations, see Table 2 for more details). For the Spearman's correlation scores, only results with a p-value under 0.05 are reported.</figcaption></figure>
<table><caption>Table 4: Retrieval at 1 (A@1) when using the F1 score to match graphs with their corresponding text on 100 selected examples from the 7L-Auto dataset. NB: Since there is no Breton training data, the MonoLR score for Breton is computed with it's closest language (Welsh).</caption><thead><tr><th rowspan="2"></th><th colspan="7">A@1</th></tr><tr><th>Breton</th><th>English</th><th>Irish</th><th>Maltese</th><th>Russian</th><th>Welsh</th><th>Xhosa</th></tr></thead><tbody><tr><th>DQE</th><td>0.53</td><td>0.95</td><td>0.32</td><td>0.48</td><td>0.05</td><td>0.39</td><td>0.38</td></tr><tr><th>FS</th><td>0.37</td><td>0.99</td><td>0.36</td><td>0.45</td><td>0.11</td><td>0.46</td><td>0.24</td></tr><tr><th>NB</th><td>0.56</td><td>0.79</td><td>0.49</td><td>0.60</td><td>0.70</td><td>0.60</td><td>0.68</td></tr><tr><th>MultiFF</th><td>0.92</td><td><strong>1.00</strong></td><td>0.84</td><td><strong>0.96</strong></td><td>0.96</td><td>0.95</td><td><strong>0.99</strong></td></tr><tr><th>MultiLR</th><td>0.93</td><td>0.99</td><td>0.85</td><td>0.94</td><td>0.93</td><td>0.97</td><td>0.98</td></tr><tr><th>MonoLR</th><td><strong>0.94</strong></td><td><strong>1.00</strong></td><td><strong>0.91</strong></td><td><strong>0.96</strong></td><td><strong>0.99</strong></td><td><strong>1.00</strong></td><td><strong>0.99</strong></td></tr></tbody></table>

The exact numbers as well as a breakdown by Precision and Recall (when possible), can be found in Table 7, Table 8, and Table 9 (Appendix D).

**Correlation is highest for 2017 Data.** Different from our 4L-RP-Human dataset, the human judgments collected during the WebNLG campaigns do not directly target precision and recall: WebNLG 2017 targets semantic faithfulness, WebNLG 2020 targets three semantic criteria related to but not identical to precision and recall, and WebNLG 2023 focuses on omissions and additions but only return a binary score no matter how much omission/addition occurs in the generated text. As explained in Section 4.2, we use the available human scores to approximate an F1 score and compute correlation between these derived F1 scores and each evaluated metric. We hypothesize that the higher correlation obtained for the 2017 data results from the fact that for this campaign, the single score provided by the human evaluation is more directly related to the unique score provided by the baseline metrics and to our F1 score. Conversely the lower correlation scores obtained by our models on the WebNLG 2020 and 2023 datasets compared to those obtained when evaluating on the 4L-RP-Human dataset are likely caused by the need to "re-construct" an F1 score from the human judgments provided by these datasets (product of three criteria for 2020 and Harmonic mean of binary scores for lack of addition and omission for 2023).

**An improvement over the state-of-the-art.** In English, our MonoLR model outperforms the three baselines despite these being optimized for this language. For the other 4 languages, the gap with these monolingual metrics is particularly pronounced. Surprisingly, for Russian the NB model is on par with our MonoLR model. This highlights the impact of using a multilingual model as base model even when fine tuning on English only. However, the low results of the NB model on the other language

shows that using NLI only, without fine tuning on task specific data does not suffice.

## 5.3 Accuracy on Retrieval.

Table 4 shows the Retrieval at 1 (A@1) for text/graph retrieval. In all languages, our fine-tuned models outperform the baselines, with the MonoLR model obtaining almost perfect scores in most of them and even outperforming FS in English. While the retrieval corpus is admittedly limited in size (10K possible combinations), the results demonstrate the effect of our approach on multilingual graph/text representation learning: for all languages, our models successfully identify the matching text given a graph and vice versa.

# 6 Conclusion

Previous work on reference less evaluation of KG-to-Text generation has mainly focused on English, providing global metrics for semantic faithfulness. We extend this work by presenting models which support the reference less evaluation of multilingual KG-to-Text generation while allowing for a finer-grained evaluation in terms of precision, recall and F1. The proposed models show strong correlation with human judgments of precision and recall for several languages, moderate to strong correlation with automatic metrics and high retrieval accuracy. On a small data set of 10K (graph-text) pairs retrieval accuracy is high indicating that the representations learned by our model provide a good basis for identifying matching KG/text pairs.

# 7 Acknowledgments

We thank the anonymous reviewers for their feedback. We gratefully acknowledge the support of the French National Research Agency (Gardent; award ANR-20-CHIA-0003, XNLG "Multilingual, Multi-Source Text Generation"). Experiments presented in this paper were carried out using the Grid'5000
testbed, supported by a scientific interest group
hosted by Inria and including CNRS, RENATER
and several Universities as well as other organi-
zations (see https://www.grid5000.fr). This
work was also granted access to the HPC resources
of IDRIS under the allocation AD0110165612
made by GENCI.

8 Limitations

First and foremost we were limited by the availabil-
ity of high quality English KG-to-Text data. While
we could generate infinite synthetic errors our orig-
inal source of correct English lexicalizations was
short and could not be securely extended.

We were limited by available multilingual data.
While machine translation has advanced signifi-
cantly over time and we attempted to filter out bad
translations, there is still a risk of noise getting into
the training data; particularly on low-resource lan-
guages. Furthermore, the lack of adequate testing
data means that, while we believe it is possible to
apply this framework to many more languages, we
were unable to put it to the test.

Our trained models are limited to the languages
we trained them for and are still prone to mistake
in certain cases, particularly when dealing with
the grammatical and morphological nuances of lan-
guages.

9 Ethical Considerations

Expanding the toolbox of evaluation metrics to
languages other than English can help democratize
and expand access to new technologies to a larger
and more diverse group of people; however, it is
important to keep in mind the limitations of these
systems. Neural Models are, at the end of the day,
statistical models and as such they are prone to
error.

It is important to always consider the multiple
layers of bias and noise involved in the creation of
these models. Every step of the way is a possible
source of bias; from the selected pretrained model
to the fine-tuning dataset, going through all the in-
termediary processes like synthetic data generation
and machine translations.

In a time of quick changes it is more important
than ever to have multiple and diverse evaluation
metrics that provide different insights into the gen-
eration of our models and remember that each and
everyone of those metrics has its own pros and cons.
We can not blindly trust a single specific metric but

instead understand and use a multitude of them to
gain a complete perspective.

**Supplementary Materials Availability:** All the
code, data, and final models produced are made
publicly available in the following repository: https://gitlab.inria.fr/wsotomar/semanti
c-evaluation-of-multilingual-data-to-t
ext-generation-via-nli-fine-tuning.
References

Thiago Castro Ferreira, Claire Gardent, Nikolai Ilinykh, Chris van der Lee, Simon Mille, Diego Moussallem, and Anastasia Shimorina. 2020. The 2020 bilingual, bi-directional WebNLG+ shared task: Overview and evaluation results (WebNLG+ 2020). In Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+), pages 55–76, Dublin, Ireland (Virtual). Association for Computational Linguistics.

Yanran Chen and Steffen Eger. 2023. MENLI: Robust evaluation metrics from natural language inference. Transactions of the Association for Computational Linguistics, 11:804–825.

Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre-training text encoders as discriminators rather than generators. In ICLR.

Liam Cripwell, Anya Belz, Claire Gardent, Albert Gatt, Claudia Borg, Marthese Borg, John Judge, Michela Lorandi, Anna Nikiforovskaya, and William Soto Martinez. 2023. The 2023 WebNLG shared task on low resource languages. overview and evaluation results (WebNLG 2023). In Proceedings of the Workshop on Multimodal, Multilingual Natural Language Generation and Multilingual WebNLG Challenge (MM-NLG 2023), pages 55–66, Prague, Czech Republic. Association for Computational Linguistics.

Ondřej Dušek and Zdeněk Kasner. 2020. Evaluating semantic accuracy of data-to-text generation with natural language inference. In Proceedings of the 13th International Conference on Natural Language Generation, pages 131–137, Dublin, Ireland. Association for Computational Linguistics.

Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language-agnostic BERT sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 878–891, Dublin, Ireland. Association for Computational Linguistics.

Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378.

Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. The WebNLG challenge: Generating text from RDF data. In Proceedings of the 10th International Conference on Natural Language Generation, pages 124–133, Santiago de Compostela, Spain. Association for Computational Linguistics.

Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. Preprint, arXiv:2111.09543.

Hassan Kane, Muhammed Yusuf Kocyigit, Ali Abdalla, Pelkins Ajanoh, and Mohamed Coulibali. 2020. NUBIA: NeUral based interchangeability assessor for text generation. In Proceedings of the 1st Workshop on Evaluating NLG Evaluation, pages 28–37, Online (Dublin, Ireland). Association for Computational Linguistics.

Amir Hossein Kargaran, Ayyoob Imani, François Yvon, and Hinrich Schuetze. 2023. GlotLID: Language identification for low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6155–6218, Singapore. Association for Computational Linguistics.

Moritz Laurer, Wouter van Atteveldt, Andreu Salleras Casas, and Kasper Welbers. 2022. Less Annotating, More Classifying – Addressing the Data Scarcity Issue of Supervised Machine Learning with Deep Transfer Learning and BERT - NLI. Preprint. Publisher: Open Science Framework.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.

Francois Meyer and Jan Buys. 2024. Triples-to-isiXhosa (T2X): Addressing the challenges of low-resource agglutinative data-to-text generation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 16841–16854, Torino, Italia. ELRA and ICCL.

Anna Nikiforovskaya and Claire Gardent. 2024. Evaluating RDF-to-text generation models for English and Russian on out of domain data. In Proceedings of the 17th International Natural Language Generation Conference, pages 134–144, Tokyo, Japan. Association for Computational Linguistics.

Joseph Olive. 2005. Global autonomous language exploitation (gale). DARPA/IPTO Proposer Information Pamphlet.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.

Maja Popović. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, pages 612–618, Copenhagen, Denmark. Association for Computational Linguistics.

Clement Rebuffel, Thomas Scialom, Laure Soulier, Benjamin Piwowarski, Sylvain Lamprier, Jacopo Stianiano, Geoffrey Scoutheeten, and Patrick Gallinari. 2021. Data-QuestEval: A referenceless metric for data-to-text semantic evaluation. In Proceedings of
the 2021 Conference on Empirical Methods in Natu-
ral Language Processing, pages 8029–8036, Online
and Punta Cana, Dominican Republic. Association
for Computational Linguistics.

Nils Reimers and Iryna Gurevych. 2019. Sentence-
BERT: Sentence embeddings using Siamese BERT-
networks. In Proceedings of the 2019 Conference on
Empirical Methods in Natural Language Processing
and the 9th International Joint Conference on Natu-
ral Language Processing (EMNLP-IJCNLP), pages
3982–3992, Hong Kong, China. Association for Com-
putational Linguistics.

Anastasia Shimorina, Claire Gardent, Shashi Narayan, and Laura Perez-Beltrachini. 2018. WebNLG Challenge: Human Evaluation Results. Technical report, Loria & Inria Grand Est.

NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation. Preprint, arXiv:2207.04672.

Huajian Zhang and Laura Perez-Beltrachini. 2024. Leveraging entailment judgements in cross-lingual summarisation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 14481–14497, Bangkok, Thailand. Association for Computational Linguistics.

Kun Zhang, Oana Balalau, and Ioana Manolescu. 2023. FactSpotter: Evaluating the factual faithfulness of graph-to-text generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10025–10042, Singapore. Association for Computational Linguistics.

Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.
# A Training Data Creation: Expanded

## A.1 Quadruple Collection

As stated before, we begin our data creation process with the English WebNLG V3.0 dataset (Castro Ferreira et al., 2020)<fnref n="14" />. This dataset contains aligned (graph $g_i$, text $t_i$) pairs. We can also refer to graph $g_i$ as $g_{ti}$, meaning it is the graph aligned with $t_i$. In the dataset, the graphs were extracted from DBPedia,<fnref n="15" /> and the texts were either automatically lexicalized or mined from Wikipedia<fnref n="16" /> before being aligned with each other by human annotators. Since graph and text are aligned the pair has a precision and recall scores of 1, forming a quadruple (graph $g_{ti}$, text $t_i$, precision=1, recall=1).

We can create variations of these original quadruples with diverse precision and recall scores by finding pairs (graph $g_j$, text $t_i$) with different levels of information overlap ($o$). To do so, we propose to keep the text static and change the graph, since it is much easier to work with and manipulate data in graph representation. For example, measuring $o$ is much easier when both elements are in graph representation, since we can just compute the intersection between both sets of triples ($o = |g_j \cap g_{ti}|$). Because of that, for most of the creation process we work with (graph $g_j$, graph $g_{ti}$) pairs instead of (graph $g_j$, text $t_i$) pairs. Only at the end of the process we substitute $g_{ti}$ back with the original text $t_i$.

Starting with our graph $g_{ti}$, if we wish to obtain a variation quadruples with precision $p$ and recall $r$, we need to find a new $g_j$ such that the following equations are true:

$$ \bullet o/|g_{ti}| = p $$

$$ \bullet o/|g_j| = r $$

At first, we look for such a $g_j$ in the list of all original graphs from WebNLG and all its subgraphs. If finding a matching graph is impossible, we create a synthetic one that satisfies the criteria.

## A.2 Synthetic Graph Creation

Given a graph $g_{ti}$, we can create a synthetic graph $g_j$ with precision $p$ and recall $r$ by first taking $o$ triples from $g_{ti}$ and then adding external triples to $g_j$ until $o/|g_j| = r$.

External triples can be procured by selecting a triple from some graph $g_k$ that does no overlap with $g_{ti}$ or by corrupting real triples from $g_{ti}$ so that the information they represent does not match the original graph.

When corrupting a real triple, we can swap the order of the elements in the triple or substitute some or all of its elements with incorrect values. When doing so, we use logical substitutions. For example, to corrupt the triple (Alan Bean | birth-Place | Wheeler, Texas), we can substitute the object Wheeler, Texas with a different value. In such a case, we would select a value that can be paired with the property birthPlace, like Miami, Florida, instead of a random value like 1932-03-15.

## A.3 Toy Example

If we start with the following dataset of aligned (graph $g_i$, text $t_i$) pairs:

<table><thead><tr><td>ID</td><td>Graph</td><td>Text</td></tr></thead><tbody><tr><td>1</td><td>Alice | occupation | Writer</td><td>Alice is a writer.</td></tr><tr><td>2</td><td>Alice | occupation | Writer<br>Alice | country | USA</td><td>Alice is an<br>American writer.</td></tr><tr><td>3</td><td>Alice | country | USA<br>Bob | country | USA</td><td>Alice and Bob are<br>Americans.</td></tr></tbody></table>

We can assign all of them precision and recall values of 1 and turn them into quadruples.

<table><thead><tr><td>ID</td><td>Graph</td><td>Text</td><td>P</td><td>R</td></tr></thead><tbody><tr><td>1</td><td>Alice | occupation | Writer</td><td>Alice is a writer.</td><td>1.00</td><td>1.00</td></tr><tr><td>2</td><td>Alice | occupation | Writer<br>Alice | country | USA</td><td>Alice is an<br>American writer.</td><td>1.00</td><td>1.00</td></tr><tr><td>3</td><td>Alice | country | USA<br>Bob | country | USA</td><td>Alice and Bob are<br>Americans.</td><td>1.00</td><td>1.00</td></tr></tbody></table>

To create new quadruples we can start by pairing texts with subgraphs or supergraphs of their original graph. For example, $g_1$ is a subgraph of $g_2$ (and therefore $g_2$ is a supergraph of $g_1$). Pairing a text with a supergraph will produce a quadruple where the text is missing information (omission), leading to a lower recall. Pairing a text with a subgraph will produce a quadruple where the text has extra information (addition/hallucination), leading to a lower precision:

<table><thead><tr><td>ID</td><td>Graph</td><td>Text</td><td>P</td><td>R</td></tr></thead><tbody><tr><td>1</td><td>Alice | occupation | Writer</td><td>Alice is a writer.</td><td>1.00</td><td>1.00</td></tr><tr><td>2</td><td>Alice | occupation | Writer<br>Alice | country | USA</td><td>Alice is an<br>American writer.</td><td>1.00</td><td>1.00</td></tr><tr><td>3</td><td>Alice | country | USA<br>Bob | country | USA</td><td>Alice and Bob are<br>Americans.</td><td>1.00</td><td>1.00</td></tr><tr><td>4</td><td>Alice | occupation | Writer</td><td>Alice is an<br><b>American</b> writer.</td><td>0.50</td><td>1.00</td></tr><tr><td>5</td><td>Alice | occupation | Writer<br><b>Alice | country | USA</b></td><td>Alice is a writer.</td><td>1.00</td><td>0.50</td></tr></tbody></table>

We can also pair a text with partially overlapping graphs. For example, $g_2$ and $g_3$ overlap in

<footnote n="14">¹⁴https://gitlab.com/shimorina/webnlg-dataset /-/tree/master/release_v3.0</footnote>

<footnote n="15">¹⁵https://www.dbpedia.org/</footnote>

<footnote n="16">¹⁶https://www.wikipedia.org/</footnote>
one triple, by matching their texts and graphs we will have quadruples where both recall and precision can be affected (there are both omissions and additions/hallucinations):

<table><thead><tr><th>ID</th><th>Graph</th><th>Text</th><th>P</th><th>R</th></tr></thead><tbody><tr><td>1</td><td>Alice | occupation | Writer</td><td>Alice is a writer.</td><td>1.00</td><td>1.00</td></tr><tr><td>2</td><td>Alice | occupation | Writer<br>Alice | country | USA</td><td>Alice is an<br>American writer.</td><td>1.00</td><td>1.00</td></tr><tr><td>3</td><td>Alice | country | USA<br>Bob | country | USA</td><td>Alice and Bob are<br>Americans.</td><td>1.00</td><td>1.00</td></tr><tr><td>4</td><td>Alice | occupation | Writer</td><td>Alice is an<br><b>American</b> writer.</td><td>0.50</td><td>1.00</td></tr><tr><td>5</td><td>Alice | occupation | Writer<br><b>Alice | country | USA</b></td><td>Alice is a writer.</td><td>1.00</td><td>0.50</td></tr><tr><td>6</td><td>Alice | country | USA<br><b>Bob | country | USA</b></td><td>Alice is an<br>American writer.</td><td>0.50</td><td>0.50</td></tr><tr><td>7</td><td><b>Alice | occupation | Writer</b><br>Alice | country | USA</td><td>Alice and Bob are<br>Americans.</td><td>0.50</td><td>0.50</td></tr></tbody></table>

Finally, we can produce new quadruples by creating synthetic graphs, either by corrupting original triples or by adding new ones:

<table><thead><tr><th>ID</th><th>Graph</th><th>Text</th><th>P</th><th>R</th></tr></thead><tbody><tr><td>1</td><td>Alice | occupation | Writer</td><td>Alice is a writer.</td><td>1.00</td><td>1.00</td></tr><tr><td>2</td><td>Alice | occupation | Writer<br>Alice | country | USA</td><td>Alice is an<br>American writer.</td><td>1.00</td><td>1.00</td></tr><tr><td>3</td><td>Alice | country | USA<br>Bob | country | USA</td><td>Alice and Bob are<br>Americans.</td><td>1.00</td><td>1.00</td></tr><tr><td>4</td><td>Alice | occupation | Writer</td><td>Alice is an<br><b>American</b> writer.</td><td>0.50</td><td>1.00</td></tr><tr><td>5</td><td>Alice | occupation | Writer<br><b>Alice | country | USA</b></td><td>Alice is a writer.</td><td>1.00</td><td>0.50</td></tr><tr><td>6</td><td>Alice | country | USA<br><b>Bob | country | USA</b></td><td>Alice is an<br>American writer.</td><td>0.50</td><td>0.50</td></tr><tr><td>7</td><td><b>Alice | occupation | Writer</b><br>Alice | country | USA</td><td>Alice and Bob are<br>Americans.</td><td>0.50</td><td>0.50</td></tr><tr><td>8</td><td>Alice | occupation | Writer<br>Alice | country | <b>Mexico</b></td><td>Alice is an<br><b>American</b> writer.</td><td>0.50</td><td>0.50</td></tr><tr><td>9</td><td>Alice | occupation | Writer<br>Alice | country | USA<br><b>Alice | birthDate | 2000-01-01</b></td><td>Alice is an<br>American writer.</td><td>1.00</td><td>0.66</td></tr></tbody></table>
## B Training Hyper Parameters

<table><caption>Table 5: Training hyperparameters of all our models. *The model is fine-tuned on 6 languages together. **The model is fine-tuned in only 1 language.</caption><thead><tr><th>Parameter</th><th>MultiFF*</th><th>MultiLR*</th><th>MonoLR**</th></tr></thead><tbody><tr><td>Training Hardware</td><td>1 32GB V100</td><td>1 32GB V100</td><td>1 32GB V100</td></tr><tr><td>Training Instances</td><td>~ 3 544 994</td><td>~ 3 544 994</td><td>~ 590 832</td></tr><tr><td>Training Epochs</td><td>1</td><td>1</td><td>1</td></tr><tr><td>Training Time</td><td>~ 7h</td><td>~ 11h</td><td>~ 2h</td></tr><tr><td>Warmup Steps</td><td>10%</td><td>10%</td><td>10%</td></tr><tr><td>Scheduler</td><td>WarmupLinear</td><td>WarmupLinear</td><td>WarmupLinear</td></tr><tr><td>Optimizer</td><td>AdamW</td><td>AdamW</td><td>AdamW</td></tr><tr><td>Learning Rate</td><td>2e-5</td><td>2e-5</td><td>2e-5</td></tr><tr><td>Loss Function</td><td>BCELoss</td><td>BCELoss</td><td>BCELoss</td></tr><tr><td>Rank</td><td>N/A</td><td>N/A</td><td>32</td></tr><tr><td>Total parameters</td><td>278 811 651</td><td>283 507 299</td><td>283 507 299</td></tr><tr><td>Trained parameters</td><td>278 811 651</td><td>5 382 240</td><td>5 382 240</td></tr></tbody></table>
C 7L-Auto Results

<table><thead><tr><th rowspan="2"></th><th colspan="5">Breton</th><th colspan="5">English</th></tr><tr><th>BLEU ↑</th><th>ChrF++ ↑</th><th>~TER ↑</th><th>BERTScore ↑</th><th>SBERT ↑</th><th>BLEU ↑</th><th>ChrF++ ↑</th><th>~TER ↑</th><th>BERTScore ↑</th><th>SBERT ↑</th></tr></thead><tbody><tr><th>DQE</th><td>0.24</td><td>0.30</td><td><strong>0.18</strong></td><td>0.31</td><td>0.35</td><td><strong>0.51</strong></td><td><strong>0.60</strong></td><td>0.50</td><td><strong>0.62</strong></td><td><strong>0.68</strong></td></tr><tr><th>FS</th><td>0.25</td><td>0.28</td><td>—</td><td>0.29</td><td>0.32</td><td><strong>0.51</strong></td><td><strong>0.60</strong></td><td>0.46</td><td>0.61</td><td>0.67</td></tr><tr><th>NB</th><td>—</td><td>—</td><td>—</td><td>-0.09</td><td>—</td><td>-0.27</td><td>-0.30</td><td>-0.39</td><td>-0.36</td><td>-0.30</td></tr><tr><th>MultiFF</th><td>0.37</td><td>0.41</td><td>0.12</td><td>0.39</td><td>0.34</td><td>0.36</td><td>0.47</td><td>0.41</td><td>0.48</td><td>0.54</td></tr><tr><th>MultiLR</th><td>0.43</td><td><strong>0.52</strong></td><td><strong>0.18</strong></td><td>0.47</td><td><strong>0.41</strong></td><td>0.40</td><td>0.53</td><td>0.47</td><td>0.54</td><td>0.60</td></tr><tr><th>MonoLR</th><td><strong>0.45</strong></td><td>0.50</td><td>0.16</td><td><strong>0.49</strong></td><td>0.39</td><td>0.44</td><td>0.58</td><td><strong>0.53</strong></td><td>0.61</td><td>0.67</td></tr></tbody></table>

<table><thead><tr><th rowspan="2"></th><th colspan="5">Irish</th><th colspan="5">Maltese</th></tr><tr><th>BLEU ↑</th><th>ChrF++ ↑</th><th>F1 ↑</th><th>BERTScore ↑</th><th>SBERT ↑</th><th>BLEU ↑</th><th>ChrF++ ↑</th><th>F1 ↑</th><th>BERTScore ↑</th><th>SBERT ↑</th></tr></thead><tbody><tr><th>DQE</th><td>0.23</td><td>0.21</td><td>0.12</td><td>0.17</td><td>0.23</td><td>0.54</td><td>0.60</td><td>0.37</td><td>0.55</td><td>0.60</td></tr><tr><th>FS</th><td>0.29</td><td>0.31</td><td>0.17</td><td>0.28</td><td><strong>0.33</strong></td><td>0.60</td><td>0.66</td><td>0.38</td><td>0.62</td><td>0.62</td></tr><tr><th>NB</th><td>-0.12</td><td>-0.11</td><td>-0.18</td><td>-0.21</td><td>-0.07</td><td>0.08</td><td>0.10</td><td>—</td><td>0.02</td><td>0.14</td></tr><tr><th>MultiFF</th><td>0.28</td><td>0.29</td><td>0.14</td><td>0.29</td><td>0.21</td><td>0.70</td><td>0.78</td><td>0.46</td><td>0.72</td><td>0.74</td></tr><tr><th>MultiLR</th><td>0.38</td><td>0.40</td><td>0.20</td><td>0.39</td><td>0.27</td><td>0.72</td><td>0.80</td><td><strong>0.49</strong></td><td>0.74</td><td><strong>0.78</strong></td></tr><tr><th>MonoLR</th><td><strong>0.40</strong></td><td><strong>0.41</strong></td><td><strong>0.22</strong></td><td><strong>0.41</strong></td><td>0.29</td><td><strong>0.76</strong></td><td><strong>0.84</strong></td><td><strong>0.49</strong></td><td><strong>0.77</strong></td><td><strong>0.78</strong></td></tr></tbody></table>

<table><thead><tr><th rowspan="2"></th><th colspan="5">Russian</th><th colspan="5">Welsh</th></tr><tr><th>BLEU ↑</th><th>ChrF++ ↑</th><th>F1 ↑</th><th>BERTScore ↑</th><th>SBERT ↑</th><th>BLEU ↑</th><th>ChrF++ ↑</th><th>F1 ↑</th><th>BERTScore ↑</th><th>SBERT ↑</th></tr></thead><tbody><tr><th>DQE</th><td>-0.05</td><td>-0.07</td><td>-0.03</td><td>-0.08</td><td>-0.08</td><td>0.34</td><td>0.37</td><td>0.29</td><td>0.37</td><td>0.47</td></tr><tr><th>FS</th><td>-0.02</td><td>—</td><td>0.02</td><td>—</td><td>0.04</td><td>0.36</td><td>0.40</td><td>0.29</td><td>0.41</td><td>0.47</td></tr><tr><th>NB</th><td>-0.07</td><td>-0.12</td><td>-0.22</td><td>-0.22</td><td>-0.07</td><td>-0.09</td><td>-0.09</td><td>-0.16</td><td>-0.19</td><td>-0.07</td></tr><tr><th>MultiFF</th><td>0.13</td><td>0.19</td><td>0.11</td><td>0.18</td><td>0.23</td><td>0.46</td><td>0.49</td><td>0.32</td><td>0.49</td><td>0.44</td></tr><tr><th>MultiLR</th><td><strong>0.26</strong></td><td><strong>0.37</strong></td><td><strong>0.29</strong></td><td><strong>0.39</strong></td><td><strong>0.35</strong></td><td><strong>0.54</strong></td><td><strong>0.59</strong></td><td><strong>0.39</strong></td><td><strong>0.59</strong></td><td><strong>0.51</strong></td></tr><tr><th>MonoLR</th><td>0.25</td><td>0.36</td><td>0.28</td><td>0.38</td><td>0.34</td><td>0.53</td><td>0.58</td><td>0.37</td><td><strong>0.59</strong></td><td><strong>0.51</strong></td></tr></tbody></table>

<table><caption>Table 6: Spearman's Correlation (ρ) of the F1 score from different automatic metrics against classic reference-based metrics on the 7L-Auto dataset. Only results with a p-value under 0.05 are reported. NB: Since there is no Breton training data, the MonoLR score for Breton is computed with it's closest language (Welsh).</caption><thead><tr><th rowspan="2"></th><th colspan="5">Xhosa</th></tr><tr><th>BLEU ↑</th><th>ChrF++ ↑</th><th>F1 ↑</th><th>BERTScore ↑</th><th>SBERT ↑</th></tr></thead><tbody><tr><th>DQE</th><td>-0.10</td><td>-0.04</td><td>-0.14</td><td>-0.11</td><td>-0.12</td></tr><tr><th>FS</th><td>0.19</td><td>0.18</td><td>0.18</td><td>0.11</td><td>0.13</td></tr><tr><th>NB</th><td>-0.25</td><td>-0.27</td><td>-0.30</td><td>-0.24</td><td>-0.26</td></tr><tr><th>MultiFF</th><td>—</td><td>-0.05</td><td>-0.11</td><td>—</td><td>-0.05</td></tr><tr><th>MultiLR</th><td>0.19</td><td>0.32</td><td>0.15</td><td>0.22</td><td>0.21</td></tr><tr><th>MonoLR</th><td><strong>0.22</strong></td><td><strong>0.34</strong></td><td><strong>0.19</strong></td><td><strong>0.26</strong></td><td><strong>0.25</strong></td></tr></tbody></table>
D WebNLG Human Judgment Results

<table><caption>Table 7: Root Mean Squared Error (RMSE) and Spearman's correlation (ρ) of the F1 score from different automatic metrics against the *English WebNLG 2017* human annotations. For the Spearman's correlation scores, only results with a p-value under 0.05 are reported.</caption><thead><tr><th colspan="3">English</th></tr><tr><th></th><th colspan="2">F1</th></tr><tr><th></th><th>RMSE ↓</th><th>ρ↑</th></tr></thead><tbody><tr><th>DQE</th><td>0.29</td><td>0.65</td></tr><tr><th>FS</th><td>0.29</td><td>0.71</td></tr><tr><th>NB</th><td>0.30</td><td>0.69</td></tr><tr><th>MultiFF</th><td>0.33</td><td>0.64</td></tr><tr><th>MutiLR</th><td><strong>0.26</strong></td><td>0.70</td></tr><tr><th>MonoLR</th><td><strong>0.26</strong></td><td><strong>0.73</strong></td></tr></tbody></table>

<table><caption>Table 8: Root Mean Squared Error (RMSE) and Spearman's correlation (ρ) of the Precision, Recall, and F1 score from different automatic metrics against the *English and Russian WebNLG 2020* human annotations. For the Spearman's correlation scores, only results with a p-value under 0.05 are reported.</caption><thead><tr><th rowspan="2"></th><th colspan="6">English</th><th colspan="6">Russian</th></tr><tr><th colspan="2">P</th><th colspan="2">R</th><th colspan="2">F1</th><th colspan="2">P</th><th colspan="2">R</th><th colspan="2">F1</th></tr><tr><th></th><th>RMSE ↓</th><th>ρ↑</th><th>RMSE ↓</th><th>ρ↑</th><th>RMSE ↓</th><th>ρ↑</th><th>RMSE ↓</th><th>ρ↑</th><th>RMSE ↓</th><th>ρ↑</th><th>RMSE ↓</th><th>ρ↑</th></tr></thead><tbody><tr><th>DQE</th><td>0.27</td><td>0.37</td><td>0.32</td><td>0.32</td><td>0.28</td><td>0.37</td><td>0.79</td><td>—</td><td>0.84</td><td>—</td><td>0.81</td><td>—</td></tr><tr><th>FS</th><td>0.21</td><td>0.35</td><td>0.18</td><td>0.45</td><td>0.19</td><td>0.48</td><td>0.67</td><td>—</td><td>0.79</td><td>—</td><td>0.70</td><td>—</td></tr><tr><th>NB</th><td>0.28</td><td>0.28</td><td>0.16</td><td>0.43</td><td>0.24</td><td>0.41</td><td>0.30</td><td>0.21</td><td>0.20</td><td>0.39</td><td>0.27</td><td>0.39</td></tr><tr><th>MultiFF</th><td>0.22</td><td>0.22</td><td>0.15</td><td>0.38</td><td>0.18</td><td>0.34</td><td>0.25</td><td>0.14</td><td><strong>0.14</strong></td><td>0.42</td><td>0.21</td><td>0.30</td></tr><tr><th>MultiLR</th><td>0.22</td><td>0.36</td><td>0.20</td><td>0.37</td><td>0.17</td><td>0.39</td><td>0.26</td><td>0.19</td><td>0.22</td><td>0.36</td><td>0.23</td><td>0.31</td></tr><tr><th>MonoLR</th><td><strong>0.20</strong></td><td><strong>0.44</strong></td><td><strong>0.14</strong></td><td><strong>0.47</strong></td><td><strong>0.15</strong></td><td><strong>0.50</strong></td><td><strong>0.24</strong></td><td><strong>0.25</strong></td><td>0.16</td><td><strong>0.44</strong></td><td><strong>0.20</strong></td><td><strong>0.39</strong></td></tr></tbody></table>

<table><thead><tr><th rowspan="2"></th><th colspan="6">Irish</th><th colspan="6">Maltese</th></tr><tr><th colspan="2">P</th><th colspan="2">R</th><th colspan="2">F1</th><th colspan="2">P</th><th colspan="2">R</th><th colspan="2">F1</th></tr><tr><th></th><th>RMSE ↓</th><th>ρ↑</th><th>RMSE ↓</th><th>ρ↑</th><th>RMSE ↓</th><th>ρ↑</th><th>RMSE ↓</th><th>ρ↑</th><th>RMSE ↓</th><th>ρ↑</th><th>RMSE ↓</th><th>ρ↑</th></tr></thead><tbody><tr><th>DQE</th><td>0.65</td><td>0.11</td><td>0.64</td><td>0.09</td><td>0.62</td><td>0.12</td><td>0.60</td><td>0.14</td><td>0.59</td><td>0.10</td><td>0.58</td><td>0.12</td></tr><tr><th>FS</th><td>0.62</td><td>—</td><td>0.57</td><td>0.13</td><td>0.60</td><td>0.09</td><td>0.66</td><td>0.13</td><td>0.52</td><td>0.17</td><td>0.61</td><td>0.14</td></tr><tr><th>NB</th><td>0.52</td><td>0.14</td><td>0.46</td><td>0.22</td><td>0.54</td><td>0.16</td><td>0.48</td><td>0.20</td><td>0.47</td><td>0.37</td><td>0.49</td><td>0.29</td></tr><tr><th>MultiFF</th><td>0.48</td><td>0.14</td><td>0.49</td><td>0.29</td><td>0.52</td><td>0.20</td><td>0.46</td><td>0.14</td><td>0.52</td><td>0.32</td><td>0.53</td><td>0.26</td></tr><tr><th>MultiLR</th><td>0.47</td><td>0.18</td><td><strong>0.45</strong></td><td>0.35</td><td><strong>0.49</strong></td><td>0.31</td><td>0.44</td><td>0.26</td><td>0.46</td><td>0.37</td><td><strong>0.49</strong></td><td>0.38</td></tr><tr><th>MonoLR</th><td><strong>0.46</strong></td><td><strong>0.21</strong></td><td><strong>0.45</strong></td><td><strong>0.37</strong></td><td>0.50</td><td><strong>0.33</strong></td><td><strong>0.43</strong></td><td><strong>0.30</strong></td><td><strong>0.45</strong></td><td><strong>0.43</strong></td><td><strong>0.49</strong></td><td><strong>0.41</strong></td></tr></tbody></table>

<table><caption>Table 9: Root Mean Squared Error (RMSE) and Spearman's correlation (ρ) of the Precision, Recall, and F1 score from different automatic metrics against the *Irish*, *Maltese*, *Russian and Welsh WebNLG 2023* human annotations. For the Spearman's correlation scores, only results with a p-value under 0.05 are reported. *NB: The original human annotations are binary labels, which might explain the high RMSE*</caption><thead><tr><th rowspan="2"></th><th colspan="6">Russian</th><th colspan="6">Welsh</th></tr><tr><th colspan="2">P</th><th colspan="2">R</th><th colspan="2">F1</th><th colspan="2">P</th><th colspan="2">R</th><th colspan="2">F1</th></tr><tr><th></th><th>RMSE ↓</th><th>ρ↑</th><th>RMSE ↓</th><th>ρ↑</th><th>RMSE ↓</th><th>ρ↑</th><th>RMSE ↓</th><th>ρ↑</th><th>RMSE ↓</th><th>ρ↑</th><th>RMSE ↓</th><th>ρ↑</th></tr></thead><tbody><tr><th>DQE</th><td>0.85</td><td>—</td><td>0.87</td><td>—</td><td>0.82</td><td>—</td><td>0.61</td><td>—</td><td>0.58</td><td>0.11</td><td>0.56</td><td>—</td></tr><tr><th>FS</th><td>0.76</td><td>—</td><td>0.86</td><td>—</td><td>0.82</td><td>—</td><td>0.69</td><td>—</td><td>0.55</td><td>0.18</td><td>0.60</td><td>0.11</td></tr><tr><th>NB</th><td>0.45</td><td>0.17</td><td>0.34</td><td>0.28</td><td>0.45</td><td>0.25</td><td>0.54</td><td>0.13</td><td><strong>0.53</strong></td><td>0.24</td><td><strong>0.58</strong></td><td>0.13</td></tr><tr><th>MultiFF</th><td>0.42</td><td>—</td><td>0.36</td><td>0.22</td><td>0.45</td><td>0.13</td><td>0.51</td><td>—</td><td>0.58</td><td>0.26</td><td>0.61</td><td>0.18</td></tr><tr><th>MultiLR</th><td>0.39</td><td>0.24</td><td>0.35</td><td>0.28</td><td>0.42</td><td>0.28</td><td><strong>0.49</strong></td><td>0.18</td><td>0.54</td><td>0.29</td><td><strong>0.58</strong></td><td>0.26</td></tr><tr><th>MonoLR</th><td><strong>0.38</strong></td><td><strong>0.25</strong></td><td><strong>0.32</strong></td><td><strong>0.33</strong></td><td><strong>0.41</strong></td><td><strong>0.31</strong></td><td><strong>0.49</strong></td><td><strong>0.21</strong></td><td>0.54</td><td><strong>0.34</strong></td><td><strong>0.58</strong></td><td><strong>0.29</strong></td></tr></tbody></table>
E WebNLG Human Judgment Results: System-level Correlations

For comparison, we also report results obtained using the script from Zhang et al. (2023) for system level correlations.

<table><caption>Table 10: System-level correlation (Pearson's r, Spearman's ρ, and Kendall's τ) of the F1 score from different automatic metrics against the *English WebNLG 2017* human annotations. *OgFS are the results reported in the original FactSpotter paper.</caption><thead><tr><th></th><th colspan="3">English</th></tr><tr><th></th><th colspan="3">Semantic</th></tr><tr><th></th><th>r↑</th><th>ρ↑</th><th>τ↑</th></tr></thead><tbody><tr><th>OgFS*</th><td><strong>0.97</strong></td><td><strong>0.93</strong></td><td><strong>0.85</strong></td></tr><tr><th>FS</th><td>0.95</td><td>0.91</td><td>0.80</td></tr><tr><th>NB</th><td>0.96</td><td>0.91</td><td>0.80</td></tr><tr><th>MultiFF</th><td>0.90</td><td>0.82</td><td>0.66</td></tr><tr><th>MultiLR</th><td>0.93</td><td>0.84</td><td>0.70</td></tr><tr><th>MonoLR</th><td>0.96</td><td>0.92</td><td>0.81</td></tr></tbody></table>

<table><thead><tr><th></th><th colspan="9">English</th></tr><tr><th></th><th colspan="3">Correctness</th><th colspan="3">Relevance</th><th colspan="3">Data Coverage</th></tr><tr><th></th><th>r↑</th><th>ρ↑</th><th>τ↑</th><th>r↑</th><th>ρ↑</th><th>τ↑</th><th>r↑</th><th>ρ↑</th><th>τ↑</th></tr></thead><tbody><tr><th>OgFS*</th><td>0.94</td><td>0.80</td><td>0.64</td><td>0.96</td><td>0.79</td><td>0.64</td><td>0.91</td><td>0.87</td><td>0.71</td></tr><tr><th>FS</th><td><strong>0.97</strong></td><td><strong>0.84</strong></td><td><strong>0.70</strong></td><td><strong>0.98</strong></td><td><strong>0.80</strong></td><td><strong>0.65</strong></td><td><strong>0.97</strong></td><td><strong>0.94</strong></td><td><strong>0.83</strong></td></tr><tr><th>NB</th><td>0.94</td><td>0.75</td><td>0.60</td><td>0.91</td><td>0.69</td><td>0.53</td><td>0.95</td><td>0.90</td><td>0.77</td></tr><tr><th>MultiFF</th><td>0.93</td><td>0.75</td><td>0.60</td><td>0.95</td><td>0.77</td><td>0.62</td><td>0.93</td><td>0.92</td><td>0.79</td></tr><tr><th>MultiLR</th><td>0.92</td><td>0.76</td><td>0.60</td><td>0.92</td><td>0.78</td><td>0.62</td><td>0.92</td><td>0.89</td><td>0.74</td></tr><tr><th>MonoLR</th><td>0.93</td><td>0.75</td><td>0.60</td><td>0.94</td><td>0.78</td><td>0.64</td><td>0.94</td><td><strong>0.94</strong></td><td><strong>0.83</strong></td></tr></tbody></table>

<table><caption>Table 11: System-level correlation (Pearson's r, Spearman's ρ, and Kendall's τ) of the Precision and Recall score from different automatic metrics against the *English and Russian WebNLG 2020* human annotations. *OgFS are the results reported in the original FactSpotter paper.</caption><thead><tr><th></th><th colspan="9">Russian</th></tr><tr><th></th><th colspan="3">Correctness</th><th colspan="3">Relevance</th><th colspan="3">Data Coverage</th></tr><tr><th></th><th>r↑</th><th>ρ↑</th><th>τ↑</th><th>r↑</th><th>ρ↑</th><th>τ↑</th><th>r↑</th><th>ρ↑</th><th>τ↑</th></tr></thead><tbody><tr><th>FS</th><td>-0.82</td><td>0.79</td><td>—</td><td>0.62</td><td>0.63</td><td><strong>0.81</strong></td><td>—</td><td>0.83</td><td>0.75</td></tr><tr><th>NB</th><td><strong>0.80</strong></td><td><strong>0.83</strong></td><td><strong>0.75</strong></td><td><strong>0.83</strong></td><td><strong>0.85</strong></td><td>0.76</td><td>0.83</td><td>0.85</td><td>0.76</td></tr><tr><th>MultiFF</th><td>—</td><td>—</td><td>—</td><td>0.79</td><td>0.82</td><td>0.74</td><td><strong>0.95</strong></td><td>0.88</td><td>0.78</td></tr><tr><th>MultiLR</th><td>—</td><td>—</td><td>—</td><td>0.80</td><td>0.83</td><td>0.74</td><td>0.94</td><td><strong>0.89</strong></td><td>0.79</td></tr><tr><th>MonoLR</th><td>—</td><td>—</td><td>—</td><td>0.80</td><td>0.83</td><td>0.76</td><td>0.92</td><td><strong>0.89</strong></td><td><strong>0.80</strong></td></tr></tbody></table>

<table><thead><tr><th></th><th colspan="9">Irish</th></tr><tr><th></th><th colspan="3">Additions</th><th colspan="3">Omissions</th><th colspan="3">Maltese</th></tr><tr><th></th><th>r↑</th><th>ρ↑</th><th>τ↑</th><th>r↑</th><th>ρ↑</th><th>τ↑</th><th>r↑</th><th>ρ↑</th><th>τ↑</th></tr></thead><tbody><tr><th>FS</th><td>0.79</td><td>0.60</td><td>—</td><td>0.97</td><td>0.97</td><td>—</td><td>0.82</td><td>0.79</td><td>—</td></tr><tr><th>NB</th><td>0.97</td><td><strong>1.00</strong></td><td>—</td><td>0.97</td><td><strong>1.00</strong></td><td>—</td><td><strong>1.00</strong></td><td>0.99</td><td>—</td></tr><tr><th>MultiFF</th><td>0.97</td><td><strong>1.00</strong></td><td>—</td><td>0.97</td><td><strong>1.00</strong></td><td>—</td><td><strong>1.00</strong></td><td><strong>1.00</strong></td><td>—</td></tr><tr><th>MultiLR</th><td>0.97</td><td><strong>1.00</strong></td><td>—</td><td>0.96</td><td><strong>1.00</strong></td><td>—</td><td><strong>1.00</strong></td><td><strong>1.00</strong></td><td>—</td></tr><tr><th>MonoLR</th><td><strong>0.98</strong></td><td><strong>1.00</strong></td><td>—</td><td><strong>0.98</strong></td><td><strong>1.00</strong></td><td>—</td><td><strong>1.00</strong></td><td><strong>1.00</strong></td><td>—</td></tr></tbody></table>

<table><caption>Table 12: System-level correlation (Pearson's r, Spearman's ρ, and Kendall's τ) of the Precision and Recall score from different automatic metrics against the *Irish, Maltese, Russian and Welsh WebNLG 2023* human annotations.</caption><thead><tr><th></th><th colspan="9">Russian</th></tr><tr><th></th><th colspan="3">Additions</th><th colspan="3">Omissions</th><th colspan="3">Welsh</th></tr><tr><th></th><th>r↑</th><th>ρ↑</th><th>τ↑</th><th>r↑</th><th>ρ↑</th><th>τ↑</th><th>r↑</th><th>ρ↑</th><th>τ↑</th></tr></thead><tbody><tr><th>FS</th><td>-0.48</td><td>-0.43</td><td>—</td><td>0.61</td><td>0.47</td><td>—</td><td><strong>0.97</strong></td><td><strong>0.84</strong></td><td>—</td></tr><tr><th>NB</th><td><strong>0.25</strong></td><td><strong>0.15</strong></td><td>—</td><td><strong>0.90</strong></td><td><strong>0.71</strong></td><td>—</td><td><strong>0.97</strong></td><td><strong>0.84</strong></td><td>—</td></tr><tr><th>MultiFF</th><td>-0.66</td><td>-0.49</td><td>—</td><td>0.68</td><td>0.64</td><td>—</td><td><strong>0.97</strong></td><td><strong>0.84</strong></td><td>—</td></tr><tr><th>MultiLR</th><td>-0.80</td><td>-0.60</td><td>—</td><td>-0.41</td><td>-0.11</td><td>—</td><td><strong>0.97</strong></td><td><strong>0.84</strong></td><td>—</td></tr><tr><th>MonoLR</th><td>-0.48</td><td>0.05</td><td>—</td><td>0.45</td><td>0.48</td><td>—</td><td><strong>0.97</strong></td><td><strong>0.84</strong></td><td>—</td></tr></tbody></table>
## F Annotation Examples

<table><caption>Table 13: English samples from 4L-RP-Human with their Human and MonoLR scores in a scale from 0 to 1. The samples were selected from the 1st (good), 3rd (median) and 5th (bad) quintile based on the accuracy of MonoLR compared to the Human judgment.</caption><thead><tr><th rowspan="2">Sample</th><th colspan="2">Precision</th><th colspan="2">Recall</th><th rowspan="2">Quintile</th></tr><tr><th>Human</th><th>MonoLR</th><th>Human</th><th>MonoLR</th></tr></thead><tbody><tr><td><strong>Graph:</strong><br>Mermaid (Train song) | genre | Pop rock<br>Mermaid (Train song) | runtime | 3.16<br>Mermaid (Train song) | releaseDate | 2012-12-27<br>Mermaid (Train song) | precededBy | This'll Be My Year<br>Mermaid (Train song) | writer | Espen Lind<br><strong>Text:</strong><br>Mermaid is a pop rock song written by Espen Lind.<br>It was released on 27 December 2012 and has a run time of 3.16.</td><td>1.00</td><td>0.98</td><td>0.62</td><td>0.75</td><td>1st</td></tr><tr><td><strong>Graph:</strong><br>Turkey | longName | Republic of Turkey<br>Nurhan Atasoy | nationality | Turkish people<br>Nurhan Atasoy | citizenship | Turkey<br>Turkey | language | Turkish language<br><strong>Text:</strong><br>The Turkish language is spoken in Turkey where the leader is known as the Republic of Turkey. The country is the location of the Ataturk Atasoy which is a citizenship of the Turkish people.</td><td>0.19</td><td>0.32</td><td>0.25</td><td>0.37</td><td>3rd</td></tr><tr><td><strong>Graph:</strong><br>Ciudad Ayala | populationMetro | 1777539<br><strong>Text:</strong><br>1777539 is the population metro in the country.</td><td>0.12</td><td>0.88</td><td>0.25</td><td>0.52</td><td>5th</td></tr></tbody></table>

<table><caption>Table 14: Maltese samples from 4L-RP-Human with their Human and MonoLR scores in a scale from 0 to 1. The samples were selected from the 1st (good), 3rd (median) and 5th (bad) quintile based on the accuracy of MonoLR compared to the Human judgment.</caption><thead><tr><th rowspan="2">Sample</th><th colspan="2">Precision</th><th colspan="2">Recall</th><th rowspan="2">Quintile</th></tr><tr><th>Human</th><th>MonoLR</th><th>Human</th><th>MonoLR</th></tr></thead><tbody><tr><td><strong>Graph:</strong><br>McVeagh of the South Seas | director | Cyril Bruce<br>McVeagh of the South Seas | writer | Harry Carey (actor born 1878)<br><strong>Text:</strong><br>McVeagh tal-Baħar tan-Nofsinhar kien miktub minn Harry Carey (attur imwieled 1878) u<br>dirett minn Cyril Bruce.<br><strong>English MT:</strong><br>McVeagh of the Southern Seaboard was written by Harry Carey (actor born 1878) and<br>directed by Cyril Bruce.</td><td>0.92</td><td>0.96</td><td>0.83</td><td>0.88</td><td>1st</td></tr><tr><td><strong>Graph:</strong><br>United States | leaderTitle | Vice President<br>Darinka Dentcheva | residence | United States<br><strong>Text:</strong><br>Il-Viċi President huwa l-mexxej tal-Istati Uniti<br><strong>English MT:</strong><br>The Vice President is the leader of the United States.</td><td>0.75</td><td>0.99</td><td>0.42</td><td>0.67</td><td>3rd</td></tr><tr><td><strong>Graph:</strong><br>Turkey | demonym | Turk<br><strong>Text:</strong><br>Id-demonimu tal-abitanti tal-belt ta' Turkmen huwa ta<br><strong>English MT:</strong><br>The demonym of the inhabitants of the city of Turkmen is</td><td>0.08</td><td>0.84</td><td>0.08</td><td>0.71</td><td>5th</td></tr></tbody></table>
<table><caption>Table 15: Russian samples from 4L-RP-Human with their Human and MonoLR scores in a scale from 0 to 1. The samples were selected from the 1st (good), 3rd (median) and 5th (bad) quintile based on the accuracy of MonoLR compared to the Human judgment.</caption><thead><tr><th rowspan="2">Sample</th><th colspan="2">Precision</th><th colspan="2">Recall</th><th rowspan="2">Quintile</th></tr><tr><th>Human</th><th>MonoLR</th><th>Human</th><th>MonoLR</th></tr></thead><tbody><tr><td><strong>Graph:</strong><br>(66063) 1998 RO1 | meanTemperature | 265.0 (kelvins)<br>(66063) 1998 RO1 | apoapsis | 254989570.60815 (kilometres)<br>(66063) 1998 RO1 | epoch | 2013-11-04<br>(66063) 1998 RO1 | orbitalPeriod | 360.29 (days)</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><strong>Text:</strong><br>Небесное тело, известное как (66063) 1998 RO1, имеет среднюю температуру 265 Кельвивинов и орбитальный период 360,29 дней.<br>Его апоцентр - 254989570.60815 километров, а его эпоха - 13 января 2016 года.<br><strong>English MT:</strong><br>The celestial body, known as (66063) 1998 RO1, has an average temperature of 265 Kelvin and an orbital period of 360.29 days. Its apocenter is 254989570.60815 kilometers, and its epoch is January 13, 2016.</td><td>0.88</td><td>0.93</td><td>0.88</td><td>0.79</td><td>1st</td></tr><tr><td><strong>Graph:</strong><br>School of Business and Social Sciences at the Aarhus University | affiliation | European University Association<br>European University Association | headquarter | Brussels<br>School of Business and Social Sciences at the Aarhus University | established | 1928</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><strong>Text:</strong><br>Школа бизнеса и социальных наук Орхусского университета была создана в 1928 году и входит в Ассоциацию университетов Европы, штаб-квартира которой находится в Брюссельском столичном регионе.<br><strong>English MT:</strong><br>The School of Business and Social Sciences at Aarhus University was founded in 1928 and is a member of the Association of European Universities, headquartered in the Brussels-Capital Region.</td><td>0.88</td><td>0.56</td><td>1.00</td><td>0.90</td><td>3rd</td></tr><tr><td><strong>Graph:</strong><br>11 Diagonal Street | location | South Africa<br><strong>Text:</strong><br>Диагонал-стрит 11 находится в Южной Африке.<br><strong>English MT:</strong><br>Diagonal Street 11 is located in South Africa.</td><td>0.12</td><td>1.00</td><td>0.12</td><td>1.00</td><td>5th</td></tr></tbody></table>

<table><caption>Table 16: Welsh samples from 4L-RP-Human with their Human and MonoLR scores in a scale from 0 to 1. The samples were selected from the 1st (good), 3rd (median) and 5th (bad) quintile based on the accuracy of MonoLR compared to the Human judgment.</caption><thead><tr><th rowspan="2">Sample</th><th colspan="2">Precision</th><th colspan="2">Recall</th><th rowspan="2">Quintile</th></tr><tr><th>Human</th><th>MonoLR</th><th>Human</th><th>MonoLR</th></tr></thead><tbody><tr><td><strong>Graph:</strong><br>McVeagh of the South Seas | imdbId | 0004319<br>McVeagh of the South Seas | director | Cyril Bruce<br>McVeagh of the South Seas | director | Harry Carey (actor born 1878)<br>McVeagh of the South Seas | starring | Harry Carey (actor born 1878)<br>McVeagh of the South Seas | writer | Harry Carey (actor born 1878)</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><strong>Text:</strong><br>Ysgrifennodd Harry Carey (a anwyd yn 1878) McVeagh of the South Seas a cyfarwyddodd<br>Cyril Bruce. Mae gan y ddata IMDb 0004319.</td><td>0.88</td><td>0.96</td><td>0.69</td><td>0.65</td><td>1st</td></tr><tr><td><strong>English MT:</strong><br>Harry Carey (born 1878) wrote McVeagh of the South Seas and Cyril<br>Bruce directed. The IMDb data has 0004319.</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><strong>Graph:</strong><br>University of Burgundy | campus | Dijon<br>Dijon | country | France</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><strong>Text:</strong><br>Mae Prifysgol Burgundaidd yn cael ei leoli yn Dijon, Ffrainc.<br><strong>English MT:</strong><br>Burgundian University is located in Dijon, France.</td><td>1.00</td><td>0.76</td><td>0.94</td><td>0.85</td><td>3rd</td></tr><tr><td><strong>Graph:</strong><br>Bionico | dishVariation | Honey<br>Bionico | country | Mexico</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><strong>Text:</strong><br>Mae Bionico yn amrywiad dysgl o Fecsico sy'n cynnwys mêl.<br><strong>English MT:</strong><br>Bionico is a variation of a Mexican dish that includes honey.</td><td>1.00</td><td>0.36</td><td>1.00</td><td>0.34</td><td>5th</td></tr></tbody></table>
G Human Annotation Instructions

<figure><img src="image_6.png" /><figcaption>Figure 5: Part 1 of 2 from the human annotation instructions.</figcaption></figure>
<figure><img src="image_7.png" /><figcaption>Figure 6: Part 2 of 2 from the human annotation instructions.</figcaption></figure>