EuroVerdict: A Multilingual Dataset for Verdict Generation Against Misinformation
Authors: Daniel Russo, Fariba Sadeghi, Stefano Menini, Marco Guerini
Accepted at: acl25

EuroVerdict: A Multilingual Dataset for Verdict Generation Against Misinformation

Daniel Russo1,2, Fariba Sadeghi1, Stefano Menini1, Marco Guerini1
1Fondazione Bruno Kessler, Italy
2University of Trento, Italy
{drusso, fsadeghi, menini, guerini}@fbk.eu

Abstract

Misinformation is a global issue that shapes public discourse, influencing opinions and decision-making across various domains. While automated fact-checking (AFC) has become essential in combating misinformation, most work in multilingual settings has focused on claim verification rather than generating explanatory verdicts (i.e. short texts discussing the veracity of the claim), leaving a gap in AFC resources beyond English. To this end, we introduce EuroVerdict, a multilingual dataset designed for verdict generation, covering eight European languages. Developed in collaboration with professional fact-checkers, the dataset comprises claims, manually written verdicts, and supporting evidence, including fact-checking articles and additional secondary sources. We evaluate EuroVerdict with Llama-3.1-8B-Instruct on verdict generation under different settings, varying the prompt language, input article language, and training approach. Our results show that fine-tuning consistently improves performance, with models fine-tuned on original-language articles achieving the highest scores in both automatic and human evaluations. Using articles in a different language from the claim slightly lowers performance; however, pairing them with language-specific prompts improves results. Zero-shot and Chain-of-Thought setups perform worse, reinforcing the benefits of fine-tuning for multilingual verdict generation.

1 Introduction

Misinformation is increasingly pervasive in modern society (Lazer et al., 2018), with significant impacts at both societal and individual levels (Adams et al., 2023). At the individual level, misinformation can strongly influence people’s beliefs and behaviours in response to false claims (Lewandowsky et al., 2012). Well-known and evident examples are vaccine hesitancy, religious extremism, and disengagement from political participation (Bronstein et al., 2021; Booth et al., 2024; Ecker et al., 2024). At the societal level, misinformation undermines trust in media (Wagner and Boczkowski, 2019), erodes public understanding of science — particularly on critical issues like health and climate change (Lewandowsky et al., 2017) — destabilizes markets (Petratos, 2021; Kogan et al., 2023), and poses a serious threat to democracy by preventing the electorate from being accurately informed (Kuklinski et al., 2000). The 2024 Global Risks Report1 identifies misinformation and disinformation as the most severe short-term global risk for fueling polarization, civil unrest, and the erosion of democratic rights.

Over the past decade, significant efforts have been made to counter misinformation worldwide (Cazzamatta, 2024; Humprecht, 2019), as evidenced by the proliferation of fact-checking organizations and growing research interest

Figure 1: EuroVerdict dataset comprises quadruples of <CLAIM, VERDICT, FC ARTICLE, EXTRA EVIDENCE> in eight European languages: German, Greek, English, Spanish, French, Italian, Polish, and Romanian.

1https://www3.weforum.org/docs/WEF_The_Global_Risks_Report_2024.pdf
CLAIM: Donald Trump is the Time Magazine 2023 Person of the Year.
VERDICT: The claim is false. The photo is actually Time Magazine’s 2016 cover, as a Time spokesperson confirmed it. In 2023 Taylor Swift was the one named TIME’s Person of the Year.
ARTICLE URL: https://checkyourfact.com/2023/12/13/fact-check-no-donald-trump-is-not-time-magazines-2023-person-of-the-year/
EXTERNAL EVIDENCE: https://time.com/6342806/person-of-the-year-2023-taylor-swift/

CLAIM: El censo electoral ha disminuido de 37 a 35 millones de personas en las elecciones del 23 de julio
VERDICT: Los datos oficiales del INE indican que el censo electoral ha aumentado en lugar de disminuir.
ARTICLE URL: https://maldita.es/malditobulo/20230801/censo-electoral-disminuido-elecciones-2023/
EXTERNAL EVIDENCE: https://www.ine.es/prensa/elecgral_nov2019.pdf; https://resultados.generales23j.es/es/resultados/0/0/0#PARTICIPATION

CLAIM: L’Università di Yale ha sviluppato una tecnologia per vaccinare le persone a loro insaputa
VERDICT: La tecnologia sviluppata dall’università di Yale funziona in modo simile ad altri vaccini nasali già esistenti, e non ci sono prove che questo vaccino possa funzionare se diffuso nell’aria come aerosol o nebulizzato. Inoltre il prodotto non risulta essere stato testato su animali di grandi dimensioni o esseri umani.
ARTICLE URL: https://facta.news/antibufale/2023/10/02/universita-yale-vaccino-covid/
EXTERNAL EVIDENCE: https://www.reuters.com/fact-check/yale-study-did-not-develop-technology-vaccination-without-consent-2023-09-22/

Table 1: Examples of entries of EuroVerdict dataset in English, Spanish, and Italian.

(Lewandowsky and van der Linden, 2021). In particular, the field of computer science has increasingly focused on developing automated systems to detect and fact-check misleading claims online (Guo et al., 2022). Within the automated fact-checking (AFC) domain, two primary approaches have emerged: (1) verdict prediction, which involves classifying the truthfulness of a claim using binary or multi-class labels, and (2) verdict production (or generation), which generates a short explanatory text, i.e. a verdict, detailing why a claim is true or false (Kotonya and Toni, 2020a; Atanasova et al., 2020; Russo et al., 2023b).

Although research shows that providing a short verdict with few key arguments is more effective than simply labelling claims as true or false (Lewandowsky et al., 2012; Sanna and Schwarz, 2006; Lombrozo, 2007), multilingual AFC efforts have focused largely on verdict detection, leaving a notable gap in the verdict production literature. Indeed, the only multilingual resource for verdict production, RU22Fact (Zeng et al., 2024), includes texts in English, Chinese, Russian, and Ukrainian, limited to the Russia-Ukraine conflict.

In this paper, we introduce EuroVerdict, a multilingual dataset for verdict generation that includes claims, explanatory verdicts, and their supporting evidence, consisting of fact-checking articles and additional secondary sources.2 EuroVerdict spans eight European languages: German, Greek, English, Spanish, French, Italian, Polish, and Romanian (Figure 1). The dataset was developed in collaboration with European professional fact-checkers, who selected the claims, manually crafted the verdicts, and supplemented existing fact-checking articles with additional evidence. The final dataset comprises 1,642 entries (~200 per language) and covers a wide range of topics, such as health, politics, and economy (examples in Table 1). To evaluate EuroVerdict, we tested the Llama-3.1-8B-Instruct language model on the verdict generation task, exploring various input settings (English and language-specific prompts) and training approaches, including in-context learning (zero-shot and Chain-of-Thought) and fine-tuning.

Our results show that fine-tuning consistently improves performance, while the Chain-of-Thought (CoT) strategy yields the weakest results. Using translated articles3 slightly lowers performance compared to original-language sources; however, this difference remains small, demonstrating both the robustness of the Llama model in handling multilingual prompts and its ability to generate high-quality verdicts even when the supporting information is not in the same language as the claim. Notably, fine-tuning leads to significant improvements on specific languages, such as Greek,

2The dataset is publicly available at github.com/LanDFBK/EuroVerdict
3With "translated article" we refer to the article in one language translated to English to simulate the case where the available evidence is in a different language than the claim.
without degrading performance in other languages, highlighting the robustness of the Llama model in multilingual settings. The human evaluation with professional fact-checkers aligns with these findings: fine-tuned models received the highest scores, and verdicts generated from original-language articles were preferred over those generated starting from translated ones. Additionally, using language-specific prompts with translated articles improves quality compared to using English prompts, while CoT remains the least effective configuration.

2 Related Work

Since the earliest theorization of automated fact-checking (AFC; Thorne and Vlachos, 2018), researchers have devoted considerable attention to the development of systems capable of detecting potentially misleading claims (*claim detection*), retrieving reliable and trustworthy information (*evidence retrieval*), and ultimately determining whether the claims are true or false (*verdict prediction*: Guo et al., 2022). More recently, advancements in large language model (LLM) generation capabilities have enabled researchers to focus on providing explanations for why claims may be true or false. This emerging task, often referred to in the literature as *verdict production/generation* (Guo et al., 2022; Kotonya and Toni, 2020a), is hereafter referred to as *verdict generation* for consistency.

Among AFC tasks, verdict generation is particularly challenging (Atanasova et al., 2020) as it requires systems not only to select and extract the most relevant arguments for fact-checking claims but also to present them in a manner that is coherent, grammatically correct, and faithful to the context. Due to these complexities, early attention-based (Kang et al., 2024) and rule-based (Yang et al., 2019) approaches were quickly superseded by summarization methods. Specifically, end-to-end fine-tuning of transformer models - whether with extractive objectives (Atanasova et al., 2020), abstractive objectives (Kotonya and Toni, 2020a), or a combination of both (Russo et al., 2023b,a) - has proven effective in generating high-quality verdicts by summarizing fact-checking articles.

However, a major limitation of this approach is the tendency of language models to produce factual inaccuracies, commonly referred to as ‘hallucinations’ (Huang et al., 2025). Recent advances in LLM generation quality have provided a pathway to address this issue by integrating evidence retrieved from curated and reliable knowledge bases (Lewis et al., 2020). The retrieval-augmented generation approach has demonstrated promise in enhancing the factual accuracy of generated verdicts, both in multimodal (Yao et al., 2023) and text-only scenarios (Zeng and Gao, 2024; Russo et al., 2024).

Ad-hoc strategies for collecting verdict generation data have relied on synthetic data generation, such as e-FEVER (Stammbach and Ash, 2020), and journalistic sources, including LIARPLUS (Al-hindi et al., 2018), PUBHEALTH (Kotonya and Toni, 2020b), LIAR++, and FullFact (Russo et al., 2023b). For more nuanced and realistic claims in the style of social media posts, datasets such as MisinfoCorrect (He et al., 2023) and VerMouth (Russo et al., 2023a), an extension of FullFact, have incorporated emotional claims and verdicts grounded in trustworthy fact-checking articles.

Despite the global popularity of AFC, research into multilingual AFC has primarily concentrated on the task of verdict prediction (Panchendraranjan and Zubia, 2024). Efforts in this area have produced both language-specific datasets, such as those for Danish (Nørgaard and Derczynski, 2021), Chinese (Hu et al., 2022), and Arabic (Baly et al., 2018; Sheik Ali et al., 2021), as well as multilingual datasets (Gupta and Srikumar, 2021). However, datasets for verdict generation remain predominantly limited to the English language.

To the best of our knowledge, RU22Fact (Zeng et al., 2024) is the only dataset that directly addresses multilingual verdict generation. This dataset includes claims and verdicts in English, Chinese, Russian, and Ukrainian. RU22Fact was created by collecting claims related to the Russia-Ukraine conflict from fact-checking websites and credible news outlets, using fact-checking justifications as explanations for claims from the former, and summarizing and manually verifying news articles for claims from the latter.

In this paper, we take a significant step forward by introducing a novel multilingual dataset for verdict generation, encompassing claims and verdicts written in eight European languages: *German, Greek, English, Spanish, French, Italian, Polish, Romanian*. The claims, spanning a wide range of topics, were sourced from the Google Fact Check platform. Verdicts were meticulously written by professional fact-checkers, that were actively involved in the dataset’s development, ensuring its accuracy and reliability.
3 Dataset Creation

We propose the EuroVerdict, the first multilingual dataset for verdict generation comprising data written by professional fact-checkers around Europe. Hereafter, we detail the dataset creation.

3.1 Data Collection

We began by collecting pairs of claims and corresponding fact-checking (FC) articles in multiple languages from reliable fact-checking sources. Subsequently, professional fact-checkers were tasked with writing a verdict for each claim based on the information presented in the article and potentially additional external resources. We adopted a nichesourcing approach for data collection to enhance the quality of our final resource, drawing inspiration from prior works (e.g., Chung et al. (2019)). This section will provide an in-depth description of our data collection procedure. In Appendix A.2 we detailed the guidelines provided to the professional fact-checkers for the creation of the dataset.

Annotators and Languages. The data collection of quadruples <CLAIM, VERDICT, FC ARTICLE, EXTRA EVIDENCE> was performed in eight languages: German, Greek, English, Spanish, French, Italian, Polish, and Romanian. The annotation process involved two professional fact-checkers per language across the eight languages. In order to collect high-quality data, all fact-checkers are native speakers of their respective languages, have a minimum of two years of fact-checking experience, and are members of the European Fact-Checking Standards Network. They were tasked with (i) selecting claims, (ii) writing verdicts following specific guidelines while grounding the information from FC articles, and (iii) providing additional material supporting the verdict. The data collection required roughly two months.

Claim Selection Procedure. This step aimed to collect a balanced dataset of approximately 200 misinformation claims per language. To ensure that our data met fact-checking standards, we started by selecting only reliable and trusted sources. For this purpose, we resorted to gathering claims from Google Fact Check Tools4 verified websites. We selected the claims through a publisher-based5 search with the service API. In addition to the claim itself, for each language, we collected other relevant information provided by Google Fact Check Tools, including the date, the URL of the fact-checking article,6 the information about the source publisher who fact-checked the claim (name and official website), and the claims rating (e.g., true, false). We then applied a filter to the collected claims, by considering only those that were rated as FALSE, PARTLY FALSE or MISSING CONTEXT (or equivalent labels according to the language being used). If different labels (RATINGS) were used, we asked the professional annotator to select only those claims that could be mapped to one of these three categories. Furthermore, we instructed annotators to exclude, as much as possible, claims containing direct quotes from politicians to mitigate potential annotator bias. After the initial filtering and collecting roughly 200 claims per language, we began the annotation procedure.

Verdict Writing Procedure. To ensure consistency across languages and minimize dependence on annotators’ personal style or organizational affiliations, we provided annotators with guidelines for the writing of the verdicts. First, we requested that the provided verdict be a concise text of a few sentences explaining why the claim is false/partly false, etc., serving as evidence or a basis for the rating. We also asked the verdict to be as neutral as possible so as to comply with The European Code of Standards for Independent Fact-Checking Organisations (EFCSN, 2022). Furthermore, we paid particular attention to multi-modal aspects, i.e. if the verdict was discussing images or video. We required annotators to ensure that the verdict was understandable without viewing accompanying images or videos. In cases where this was not possible, such claims were discarded. Finally, we provided some examples in English as a reference, such as the one below.

CLAIM: The Pope Francis has been seen partying and drinking alcohol

VERDICT: Details in the images (that circulated online) may indicate that they have been created with AI tools. Also, it was proved that the social media profiles disseminating the content were defined as ‘meme account’.

4https://toolbox.google.com/factcheck/
5More details on the full list of publishers selected for collecting the claims can be found in Table 6 in Appendix A.1.
6Articles were scraped using either ad-hoc scrapers or the Newspaper library: https://github.com/codelucas/newspaper
External Evidence Annotation Procedure.
Along with the original fact-checking article link, we asked annotators to provide a list of links to resources comprising relevant information supporting the verdict. These links could be extracted directly from the references used in the fact-checking article itself or were provided by fact-checkers. We further specified that the relevant evidence should be presented in a text-based format, meaning that direct links to videos or images are excluded.

3.2 Dataset Description
The final dataset consists of 1,642 entries, with nearly 200 per language (see Table 2). Each entry includes the claim, the fact-checking article (along with details about its publisher), the gold verdict written by professional fact-checkers, and links to additional evidence providing relevant information for verifying the claim. Examples from EuroVerdict can be found in Table 1.

In Table 2 we provide the average length of the main components of EuroVerdict dataset. In particular, we report the average number of sentences and words for the claims, the verdicts, and the fact-checking articles

Additionally, in order to have an overview of the topic covered by EuroVerdict dataset, we performed topic modeling on the claims annotated by the fact-checkers during data collection. Using the BERTopic strategy (Grootendorst, 2022), we identified underlying themes within the data (see Figure 2 for a visualization of the resulting topics on the whole dataset). Our implementation incorporated bge-M3 (Chen et al., 2024) as a multilingual embedded model, Uniform Manifold Approximation and Projection (UMAP; McInnes et al., 2018) for dimensionality reduction, and HDBSCAN algorithm from (Campello et al., 2013) as hierarchical clustering strategy. To automatically assign cluster-specific labels to the topics, we used Llama-3.1-8B-Instruct for one-shot classification as our representation model. For technical details, as well as for language-specific topic analysis, please refer to Appendix B.

<table>
  <tr>
    <th># Items</th>
    <th># Ext. Ev.</th>
    <th>Article<br>Sent.</th>
    <th>Words</th>
    <th>Claim<br>Sent.</th>
    <th>Words</th>
    <th>Verdict<br>Sent.</th>
    <th>Words</th>
  </tr>
  <tr>
    <td>EuroVerdict</td>
    <td>1642</td>
    <td>2</td>
    <td>41</td>
    <td>904</td>
    <td>1</td>
    <td>16</td>
    <td>2</td>
    <td>35</td>
  </tr>
  <tr>
    <td>German</td>
    <td>201</td>
    <td>1</td>
    <td>54</td>
    <td>802</td>
    <td>1</td>
    <td>19</td>
    <td>2</td>
    <td>15</td>
  </tr>
  <tr>
    <td>Greek</td>
    <td>195</td>
    <td>7</td>
    <td>50</td>
    <td>968</td>
    <td>1</td>
    <td>21</td>
    <td>2</td>
    <td>45</td>
  </tr>
  <tr>
    <td>English</td>
    <td>195</td>
    <td>1</td>
    <td>29</td>
    <td>567</td>
    <td>1</td>
    <td>14</td>
    <td>2</td>
    <td>30</td>
  </tr>
  <tr>
    <td>Spanish</td>
    <td>263</td>
    <td>2</td>
    <td>19</td>
    <td>563</td>
    <td>1</td>
    <td>15</td>
    <td>1</td>
    <td>23</td>
  </tr>
  <tr>
    <td>French</td>
    <td>190</td>
    <td>1</td>
    <td>55</td>
    <td>1,585</td>
    <td>1</td>
    <td>17</td>
    <td>2</td>
    <td>44</td>
  </tr>
  <tr>
    <td>Italian</td>
    <td>202</td>
    <td>1</td>
    <td>17</td>
    <td>428</td>
    <td>1</td>
    <td>16</td>
    <td>2</td>
    <td>41</td>
  </tr>
  <tr>
    <td>Polish</td>
    <td>204</td>
    <td>2</td>
    <td>64</td>
    <td>1,112</td>
    <td>1</td>
    <td>10</td>
    <td>2</td>
    <td>32</td>
  </tr>
  <tr>
    <td>Romanian</td>
    <td>192</td>
    <td>1</td>
    <td>49</td>
    <td>1,337</td>
    <td>1</td>
    <td>14</td>
    <td>2</td>
    <td>32</td>
  </tr>
</table>

Table 2: EuroVerdict statistics. We report the total number of items (#Items) and the average count of external evidence (#Ext. Ev.), words, and sentences.

![EuroVerdict topics distribution obtained with the BERTopic topic modeling technique.](page_1012_186_410_410.png)

Figure 2: EuroVerdict topics distribution obtained with the BERTopic topic modeling technique.

The topic modeling analysis identified several macro-topics related to misinformation and public discourse (Figure 2). Political discussions were prominent, with topics such as “Spanish Politics and Elections”, “Election Fraud and Irregularities in Spain’s 23-J Elections”, and “European Union policies and regulations”. Global conflicts were also well-represented, including “Russian invasion of Ukraine”, “Israel-Palestine conflict and rocket attacks”, and “Houthi attacks on ships in the Red Sea”. Health-related misinformation emerged in topics like “COVID-19 vaccine controversy and side effects” and “Health misinformation and medical endorsements”. Additionally, the analysis captured economic and environmental themes, such as “Renewable Energy Deception”, and “Natural Climate Change vs Anthropogenic Global Warming Debate”. Finally, public political figures and institutions were frequently discussed, with topics including “Donald Trump news and controversies”, “Bill Gates and his business ventures and controversies”, “Vladimir Putin’s controversial statements and accusations”, and “Yolanda Diaz controversies and lawsuits”.
4 Experimental Design

We tested the Llama-3.1-8B-Instruct multilingual model7 on EuroVerdict for the task of verdict generation, exploring various configurations.8 Our experimental design considered three key aspects: prompt configuration, article configuration, and training setup. The first two aspects address the multilingual setting by varying the language of the prompt and input, while the third pertains to the training setup.

Prompt Configurations. We investigated whether using English prompts versus language-specific prompts would influence the model’s performance, regardless of the language of the claims and the evidence articles. In particular, we started with the following English prompt.

SYSTEM: You are an expert fact-checker. Your task is to evaluate a claim based solely on the provided context. You must strictly adhere to the following rules:
1. Base your response only on the given context; do not bring in external knowledge.
2. Respond concisely in no more than three sentences.
3. Do not reference the context or mention that you had a context in your response.
4. Match the emotional tone and communication style of the claim.
5. Respond entirely in {language}.
USER: Evaluate the following claim based on the given information.
<context> {article} </context>
Claim: "{claim}"

Subsequently, we automatically translated with Google Translate the prompt into the eight languages of EuroVerdict.

Article Configurations. Regarding the article configuration, we compared the model’s performance when given fact-checking articles in their original language versus their English translations. Specifically, the articles were automatically translated using Google Translate. For the purposes of this study, we opted to use Google Translate to automatically translate the documents. This decision was primarily driven by practical considerations: Google Translate is one of the few freely available tools capable of handling the automatic translation of long documents. While we acknowledge the potential for translation errors, this approach represented one of the few viable and cost-effective solutions available to us. Moreover, our methodological assumption aligns with a realistic deployment scenario, in which translated documents are typically provided in advance and assumed to be of sufficient quality for downstream processing. This configuration serves a dual purpose: (i) to assess the model’s ability to generalize across languages and evaluate its performance on non-English content, and (ii) to address scenarios where the only available information for a given claim is in a different language.

Training Setups. Furthermore, we explored three different training setups for Llama-3.1-8B-Instruct: in-context learning through zero-shot or Chain-of-Thought (CoT) prompting, as well as fine-tuning the model on our EuroVerdict dataset. We employed Llama-3.1-8B-Instruct based on preliminary experiments with professional fact-checkers. Details are provided in Appendix C.2.

For the zero-shot experiments, we used the aforementioned prompt in both English and the respective languages. For the CoT reasoning approach, we required the model to follow a structured analytical process, beginning with identifying the specific assertion made in the claim, extracting relevant contextual information from the article, evaluating supporting or contradicting evidence, and considering potential exceptions. This step-by-step reasoning framework ensured a thorough examination before reaching a final classification by requiring the model to assign a veracity label (TRUE or FALSE) to the claim and generate a concise justification explaining its decision. CoT was only tested with an English prompt, which is provided in Appendix C.3. As the final training setup, we evaluated the model after fine-tuning it on the EuroVerdict dataset. Specifically, we fine-tuned four different versions of the Llama model while keeping the training parameters constant: two models for the prompt configurations and two for the context input configurations. Details of the fine-tuning process are provided in Appendix C.4.

5 Results and Discussion

To automatically assess model performance, we compared the generated verdicts with the gold verdicts in terms of both lexical and semantic similarity. Lexical similarity was evaluated using ROUGE metrics (ROUGE-1 and ROUGE-L), while semantic similarity was assessed with BERTScore and cosine similarity. ROUGE metrics and BERTScore
<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="2">ROUGE-1</th>
    <th colspan="2">ROUGE-L</th>
    <th colspan="2">BERTScore</th>
    <th colspan="2">Cosine Similarity</th>
  </tr>
  <tr>
    <th>EL</th><th>LS</th>
    <th>EL</th><th>LS</th>
    <th>EL</th><th>LS</th>
    <th>EL</th><th>LS</th>
  </tr>
  <tr>
    <td rowspan="3"><i>original articles</i></td>
    <td>Zero-shot</td><td>0.325</td><td>0.311</td><td>0.238</td><td>0.229</td><td>0.734</td><td>0.727</td><td>0.669</td><td>0.642</td>
  </tr>
  <tr>
    <td>Fine-tune</td><td><b>0.387</b></td><td><b>0.393</b></td><td><b>0.312</b></td><td><b>0.315</b></td><td><b>0.760</b></td><td><b>0.761</b></td><td><b>0.698</b></td><td><b>0.707</b></td>
  </tr>
  <tr>
    <td>CoT</td><td>0.270</td><td>-</td><td>0.190</td><td>-</td><td>0.710</td><td>-</td><td>0.644</td><td>-</td>
  </tr>
  <tr>
    <td rowspan="3"><i>translated articles</i></td>
    <td>Zero-shot</td><td>0.302</td><td>0.296</td><td>0.218</td><td>0.216</td><td>0.724</td><td>0.721</td><td>0.664</td><td>0.657</td>
  </tr>
  <tr>
    <td>Fine-tune</td><td><b>0.336</b></td><td><b>0.342</b></td><td><b>0.261</b></td><td><b>0.264</b></td><td><b>0.741</b></td><td><b>0.741</b></td><td><b>0.682</b></td><td><b>0.680</b></td>
  </tr>
  <tr>
    <td>CoT</td><td>0.258</td><td>-</td><td>0.183</td><td>-</td><td>0.702</td><td>-</td><td>0.616</td><td>-</td>
  </tr>
</table>

Table 3: Averaged experimental results across all languages, presented for all experimental design configurations. Prompt types include EL (english-language prompts) and LS (language-specific prompts).

were computed using the Evaluate library;9 for cosine similarity, embeddings were generated using the paraphrase-multilingual-MiniLM-L12-v2 model from Sentence Transformers (Reimers and Gurevych, 2019).10

In Table 3, we present the results averaged across all languages, providing an overall assessment of model performance. Language-specific results are detailed in Tables 9 and 10 (Appendix D.1). In Table 5, we provide examples of the generations. The results demonstrate that fine-tuning consistently improves performance across all experimental settings. In contrast, the CoT strategy yields the weakest results, suggesting that step-by-step reasoning does not necessarily enhance fact-checking accuracy in this context. In both CoT setups the model was able to correctly determine the veracity of the claims, as shown in the table 4. Given that the model in both CoT configurations was able to assign the correct label in more than 95% of the cases, we believe this step had minimal impact on the final results and did not significantly affect verdict generation. This can be attributed to cases where the model accumulated errors throughout the reasoning steps, selected arguments different from the gold verdict, or responded in the incorrect language. Additionally, using articles in a different language from the claim results in lower scores compared to using articles in their original language. However, this performance drop is not substantial, suggesting that the model is sufficiently robust to retrieve information in one language and transfer it to another for verdict generation. Indeed, the slight decline can be attributed to the additional reasoning step required for the model, which must not only extract relevant information but also effectively “translate” the knowledge into a different language.

<table>
  <tr>
    <th>Approach</th>
    <th>TRUE</th>
    <th>FALSE</th>
  </tr>
  <tr>
    <td>CoT original articles</td>
    <td>8 (3.27%)</td>
    <td>237 (96.73%)</td>
  </tr>
  <tr>
    <td>CoT translated articles</td>
    <td>10 (4.08%)</td>
    <td>235 (95.92%)</td>
  </tr>
</table>

Table 4: CoT veracity prediction results.

Analyzing performance across specific languages (Tables 9 and 10 in Appendix D.1), we observe that fine-tuning leads to significant improvements for certain languages, such as Greek, where the increase is particularly pronounced. Notably, this enhancement does not negatively impact performance in other languages, even when training across multiple languages with distinct alphabets. This finding highlights the robustness of the Llama 3.1 model, demonstrating its ability to adapt effectively to multilingual fine-tuning without degradation in overall performance.

5.1 Human Evaluation

To better assess the quality of the generations produced , we conducted a human evaluation. For this, we enlisted a pool of professional fact-checkers who were native speakers of the languages included in EuroVerdict. Evaluators were asked to rate the quality of the generated verdicts on a scale from 1 to 5, with respect to the gold verdict, across three main dimensions: alignment ("How much is the current verdict similar to the gold verdict?"), understandability ("How much is this verdict written in an understandable way?"), soundness ("How much does the verdict sound like a well-formed verdict?").11

Specifically, we provided the evaluators with the claim and gold verdict, along with five different generated verdicts, one for each configuration (always presented in random order). These included verdicts produced in zero-shot, fine-tuning with En-

9https://huggingface.co/docs/evaluate/
10https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
11Details in Appendix D.2.
<table>
  <tr>
    <th>Claim</th>
    <th>Gold Verdict</th>
    <th>Generated Verdict</th>
  </tr>
  <tr>
    <td>Israeli PM Benjamin Netanyahu is contracted with a mutated strain of Covid</td>
    <td>The claim is false. The image was manipulated from a photo of his wife Sara visiting the victim of a 2015 attack in Jerusalem. As of March 8, 2024, there have been no official reports that Netanyahu was being treated in hospital for a Covid-19 infection.</td>
    <td>The claim is false. The picture shown of Netanyahu is digitally altered from a photo of his wife visiting the victim of a 2015 attack in Jerusalem.</td>
  </tr>
  <tr>
    <td>EN</td>
    <td></td>
    <td>The claim is false. The photo of Netanyahu hospitalised is actually a digital alteration of a photo from 2015 where his wife was visiting a hospital patient. There have been no official reports of Netanyahu’s hospitalisation for a Covid-19 infection.</td>
  </tr>
  <tr>
    <td>Tom Hanks aparece en la lista de pedófilos de Jeffrey Epstein</td>
    <td>El nombre de Tom Hanks no aparece ni en la documentación judicial desclasificada relacionada con Jeffrey Epstein ni en el registro de vuelos a la isla de Epstein.</td>
    <td>El nombre de Tom Hanks no aparece en los documentos desclasificados ni en los registros de vuelos a la isla de Epstein.</td>
  </tr>
  <tr>
    <td>ES</td>
    <td></td>
    <td>El nombre de Tom Hanks no aparece en los documentos desclasificados relacionados con Jeffrey Epstein, por lo que la afirmación es falsa.</td>
  </tr>
  <tr>
    <td>Moderne Verbrenner ziehen CO2 aus der Luft.</td>
    <td>Die Behauptung ist falsch. Ein Vebrennungsmotor stößt immer CO2 aus.</td>
    <td>Die Behauptung ist falsch. Ein mit Diesel oder Benzin betriebener Verbrennungsmotor strömt im Betrieb stets CO2 aus.</td>
  </tr>
  <tr>
    <td>DE</td>
    <td></td>
    <td>Die Behauptung ist falsch. In modernen Fahrzeugen stößt der Verbrennungsmotor immer CO2 aus.</td>
  </tr>
</table>

Table 5: Examples of generated verdicts in English, French, and German using the fine-tuned Llama model with articles in their original language. Each language includes two generations: one using the English prompt (first example) and one using the language-specific prompt (second example).

glish and language-specific prompts, and CoT with the English prompt. The two context input configurations were not evaluated simultaneously, as assessing ten items at once could have increased task complexity and potentially impacted the results. To mitigate this, we alternated between verdicts generated using original-language articles as context and those generated using their English translations.

In Figure 3 we report the human evaluation results averaged across all languages for Alignment, Understandability, and Soundness as well as the overall scores. Results show that verdicts generated by the fine-tuned model using the original article and the English prompt received the highest scores. Across all training setups, generations based on the original article were rated higher than those using translated articles, confirming that maintaining the original language improves overall quality. These preferences align with the automatic evaluation results reported in Table 3, where fine-tuning outperforms zero-shot, and models fine-tuned on original articles achieve better performance than those fine-tuned on translated articles.

Interestingly, while using translated articles led to lower scores, pairing them with language-specific prompts resulted in better evaluations compared to using them with the English prompt. Finally, the CoT strategy received the lowest scores across all configurations.

6 Conclusion

In this work, we introduced EuroVerdict, a multilingual dataset for verdict generation, covering eight European languages and developed in collaboration with professional fact-checkers. The dataset

![Bar chart showing human evaluation results for Alignment, Understandability, and Soundness across configurations: fine-tuning, zero-shot, CoT, original (OA) or translated articles (TA), and English (EN) or language-specific prompts (lang).](page_1012_1347_420_312.png)

Figure 3: Human evaluation results averaged across all languages. We report the overall score (Overall Mean) and scores for Alignment, Understandability, and Soundness across configurations: fine-tuning, zero-shot, CoT, original (OA) or translated articles (TA), and English (EN) or language-specific prompts (lang).
includes claims, verdicts, and supporting evidence, consisting of fact-checking articles and additional secondary sources.

Llama-3.1-8B-Instruct was employed to test EuroVerdict over different configurations by varying the prompt language, input article language, and training setup. Our results show that fine-tuning consistently improves performance across all settings, while the Chain-of-Thought strategy yields the weakest results. Using articles written in a different language than the claim slightly reduces performance compared to original-language sources; however, the drop remains minimal, showcasing the model’s robustness in processing multilingual inputs.

Human evaluation aligns with these findings, with fine-tuned models achieving the highest scores and verdicts generated using original-language articles being preferred over those using translations. Additionally, combining translated articles with language-specific prompts improves quality compared to using English prompts. These results highlight the effectiveness of fine-tuning and the viability of multilingual fact-checking approaches, demonstrating that high-quality verdicts can be generated even when supporting evidence is in a different language than the claim.

Limitations

Despite the remarkable capabilities of large language models, they remain prone to generating factual inaccuracies. While constraining generation by conditioning the model on fact-checking articles helps mitigate this issue, it does not eliminate it entirely. Ensuring that generated verdicts strictly adhere to the provided evidence remains an open challenge.

We acknowledge that the size of our proposed dataset is relatively small. Nevertheless, this limitation is offset by the high quality of its content: the dataset has been manually curated by professional fact-checkers from various European countries. This ensures a high degree of reliability and credibility, as each entry has been rigorously verified by domain experts.

Additionally, we recognize that the use of Google Translate for document and prompt translation may introduce inaccuracies. As previously discussed, this choice was driven by practical constraints and the assumption that, in real-world applications, reliable translations are typically provided.

Additionally, although the EuroVerdict dataset includes extra evidence beyond fact-checking articles, this work does not incorporate it into the experimental design. Generating verdicts using non-fact-checking sources presents additional complexities, as highlighted by Russo et al. (2024), and we believe this should be explored as a separate task. However, with advancements in retrieval-augmented generation (Lewis et al., 2020), this dataset represents a valuable resource for future research in this direction.

Like previous work in the field, the generative system presented in this study relies heavily on the availability of fact-checking articles, assuming that relevant knowledge is readily accessible. This dependency limits its applicability in scenarios where fact-checked information is scarce or unavailable. However, we believe that the insights provided in this work can be easily transferred to more advanced retrieval-augmented generation systems, where knowledge is not assumed to be given but is dynamically retrieved before generation.

Ethical Statement

The automation of the fact-checking process is essential in today’s society, where misinformation spreads rapidly, and manual fact-checking alone cannot keep up with the sheer volume of false or misleading claims circulating online (Guo et al., 2022). However, automatic systems are not infallible: detection models may mislabel claims, and generative models can produce factual inaccuracies. Thus, we emphasize that the systems proposed in this work are not intended to function as standalone tools or replace professional fact-checkers. Instead, we view them as supportive tools that can assist fact-checkers by accelerating the verification process and enhancing their efficiency.

Acknowledgments

This work was partly supported by: the AI4TRUST project - AI-based-technologies for trustworthy solutions against disinformation (ID: 101070190), the European Union’s CERV fund under grant agreement No. 101143249 (HATEDEMICS), the European Union’s Horizon Europe research and innovation programme under grant agreement No. 101135437 (AI-CODE).

We gratefully acknowledge the contribution of the fact-checking organizations and media partners that took part in the data collection:
ADB (Asociatia Digital Bridge), DEMAGOG (Stowarzyszenie Demagog), ELLINIKA (Astiki Mi Kerdoskopiki Etairia Kentro Katapolemis Tis Parapliroforisis / Civil Non-Profit Company Kentro Katapolemis Tis Parapliroforisis), EMS (Europejskie Media Sp. zoo), EURACTIV (Euractiv Media Network B.V.), MALDITA (Fundación Maldita.es contra la desinformación: periodismo, educación investigación y datos en nuevos formatos), SkyTG24 (Sky Italia S.r.l.)

References

Zoë Adams, Magda Osman, Christos Bechlivaniadis, and Björn Meder. 2023. (why) is misinformation a problem? Perspectives on Psychological Science, 18(6):1436–1463. PMID: 36795592.

Tariq Alhindi, Savvas Petridis, and Smaranda Muresan. 2018. Where is your evidence: Improving fact-checking by justification modeling. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 85–90, Brussels, Belgium. Association for Computational Linguistics.

Pepa Atanasova, Jakob Grue Simonsen, Christina Lloma, and Isabelle Augenstein. 2020. Generating fact checking explanations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7352–7364, Online. Association for Computational Linguistics.

Ramy Baly, Mitra Mohtarami, James Glass, Lluís Márquez, Alessandro Moschitti, and Preslav Nakov. 2018. Integrating stance detection and fact checking in a unified corpus. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 21–27, New Orleans, Louisiana. Association for Computational Linguistics.

Emily Booth, Jooyoung Lee, Marian-Andrei Rizoiu, and Hany Farid. 2024. Conspiracy, misinformation, radicalisation: understanding the online pathway to indoctrination and opportunities for intervention. Journal of Sociology, 60(2):440–457.

Michael Bronstein, Erich Kummerfeld, Angus MacDonald III, and Sophia Vinogradov. 2021. Investigating the impact of anti-vaccine news on sars-cov-2 vaccine intentions. Available at SSRN 3936927.

Ricardo J. G. B. Campello, Davoud Moulavi, and Jorge Sander. 2013. Density-based clustering based on hierarchical density estimates. In Advances in Knowledge Discovery and Data Mining, pages 160–172, Berlin, Heidelberg. Springer Berlin Heidelberg.

Regina Cazzamatta. 2024. Global misinformation trends: Commonalities and differences in topics, sources of falsehoods, and deception strategies across eight countries. New Media & Society, 0(0):14614448241268896.

Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. Preprint, arXiv:2402.03216.

Yi-Ling Chung, Elizaveta Kuzmenko, Serra Sinem Tekiroglu, and Marco Guerini. 2019. CONAN - COunter NArratives through nichesourcing: a multilingual dataset of responses to fight online hate speech. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2819–2829, Florence, Italy. Association for Computational Linguistics.

Ullrich Ecker, Jon Roozenbeek, Sander van der Linden, Li Qian Tay, John Cook, Naomi Oreskes, and Stephan Lewandowsky. 2024. Misinformation poses a bigger threat to democracy than you might think. Nature, 630(8015):29–32.

EFCSN EFCSN. 2022. European code of standards for independent fact-checking organisations.

Maarten Grootendorst. 2022. Bertopic: Neural topic modeling with a class-based tf-idf procedure. arXiv preprint arXiv:2203.05794.

Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. 2022. A survey on automated fact-checking. Transactions of the Association for Computational Linguistics, 10:178–206.

Ashim Gupta and Vivek Srikumar. 2021. X-fact: A new benchmark dataset for multilingual fact checking. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 675–682, Online. Association for Computational Linguistics.

Bing He, Mustaque Ahamad, and Srijan Kumar. 2023. Reinforcement learning-based counter-misinformation response generation: A case study of covid-19 vaccine misinformation. In Proceedings of the ACM Web Conference 2023, WWW ’23, page 2698–2709, New York, NY, USA. Association for Computing Machinery.

Xuming Hu, Zhijiang Guo, GuanYu Wu, Aiwei Liu, Lijie Wen, and Philip Yu. 2022. CHEF: A pilot Chinese dataset for evidence-based fact-checking. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3362–3376, Seattle, United States. Association for Computational Linguistics.

Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst., 43(2).
Edda Humprecht. 2019. Where ‘fake news’ flourishes: a comparison across four western democracies. Information, Communication & Society, 22(13):1973–1988.

Wan Ju Kang, Jiyoung Han, Jaemin Jung, and James Thorne. 2024. XFACT team0331 at PerspectiveArg2024: Sampling from bounded clusters for diverse relevant argument retrieval. In Proceedings of the 11th Workshop on Argument Mining (ArgMining 2024), pages 182–188, Bangkok, Thailand. Association for Computational Linguistics.

Shimon Kogan, Tobias J Moskowitz, and Marina Niessner. 2023. Social media and financial news manipulation. Review of Finance, 27(4):1229–1268.

Neema Kotonya and Francesca Toni. 2020a. Explainable automated fact-checking: A survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5430–5443, Barcelona, Spain (Online). International Committee on Computational Linguistics.

Neema Kotonya and Francesca Toni. 2020b. Explainable automated fact-checking for public health claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7740–7754, Online. Association for Computational Linguistics.

James H Kuklinski, Paul J Quirk, Jennifer Jerit, David Schwieder, and Robert F Rich. 2000. Misinformation and the currency of democratic citizenship. The Journal of Politics, 62(3):790–816.

David M. J. Lazer, Matthew A. Baum, Yochai Benkler, Adam J. Berinsky, Kelly M. Greenhill, Filippo Menczer, Miriam J. Metzger, Brendan Nyhan, Gordon Pennycook, David Rothschild, Michael Schudson, Steven A. Sloman, Cass R. Sunstein, Emily A. Thorson, Duncan J. Watts, and Jonathan L. Zittrain. 2018. The science of fake news. Science, 359(6380):1094–1096.

Stephan Lewandowsky, Ullrich K. H. Ecker, Colleen M. Seifert, Norbert Schwarz, and John Cook. 2012. Misinformation and its correction: Continued influence and successful debiasing. Psychological Science in the Public Interest, 13(3):106–131. PMID: 26173286.

Stephan Lewandowsky, Ullrich K.H. Ecker, and John Cook. 2017. Beyond misinformation: Understanding and coping with the “post-truth” era. Journal of Applied Research in Memory and Cognition, 6(4):353–369.

Stephan Lewandowsky and Sander van der Linden. 2021. Countering misinformation and fake news through inoculation and prebunking. European Review of Social Psychology, 32(2):348–384.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA. Curran Associates Inc.

Tania Lombrozo. 2007. Simplicity and probability in causal explanation. Cognitive psychology, 55(3):232–257.

Leland McInnes, John Healy, and James Melville. 2018. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426.

Jeppe Nørregaard and Leon Derczynski. 2021. DanFEVER: claim verification dataset for Danish. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pages 422–428, Reykjavik, Iceland (Online). Linköping University Electronic Press, Sweden.

Rrubaa Panchendrarajan and Arkaitz Zubiaga. 2024. Claim detection for automated fact-checking: A survey on monolingual, multilingual and cross-lingual research. Natural Language Processing Journal, 7:100066.

Pythagoras N. Petratos. 2021. Misinformation, disinformation, and fake news: Cyber risks to business. Business Horizons, 64(6):763–774. CIBER SPECIAL ISSUE: CYBERSECURITY IN CRISIS.

Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.

Daniel Russo, Shane Kaszefski-Yaschuk, Jacopo Staiano, and Marco Guerini. 2023a. Countering misinformation via emotional response generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11476–11492, Singapore. Association for Computational Linguistics.

Daniel Russo, Stefano Menini, Jacopo Staiano, and Marco Guerini. 2024. Face the facts! evaluating rag-based fact-checking pipelines in realistic settings. Preprint. arXiv:2412.15189.

Daniel Russo, Serra Sinem Tekiroğlu, and Marco Guerini. 2023b. Benchmarking the generation of fact checking explanations. Transactions of the Association for Computational Linguistics, 11:1250–1264.

Lawrence J. Sanna and Norbert Schwarz. 2006. Metacognitive experiences and human judgment: The case of hindsight bias and its debiasing. Current Directions in Psychological Science, 15(4):172–176.

Zien Sheikh Ali, Watheq Mansour, Tamer Elsayed, and Abdulaziz Al-Ali. 2021. AraFacts: The first large
Arabic dataset of naturally occurring claims. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 231–236, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.

Dominik Stammbach and Elliott Ash. 2020. e-fever: Explanations and summaries for automated fact checking. In Conference for Truth and Trust Online.

James Thorne and Andreas Vlachos. 2018. Automated fact checking: Task formulations, methods and future directions. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3346–3359, Santa Fe, New Mexico, USA. Association for Computational Linguistics.

Maria Celeste Wagner and Pablo J. Boczkowski. 2019. The reception of fake news: The interpretations and practices that shape the consumption of perceived misinformation. Digital Journalism, 7(7):870–885.

Fan Yang, Shiva K. Pentyalu, Sina Mohseni, Mengnan Du, Hao Yuan, Rhema Linder, Eric D. Ragan, Shuiwang Ji, and Xia (Ben) Hu. 2019. Xfake: Explainable fake news detector with visualizations. In The World Wide Web Conference. WWW ’19, page 3600–3604, New York, NY, USA. Association for Computing Machinery.

Barry Menglong Yao, Aditya Shah, Lichao Sun, Jin-Hee Cho, and Lifu Huang. 2023. End-to-end multimodal fact-checking and explanation generation: A challenging dataset and models. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR ’23, page 2733–2743, New York, NY, USA. Association for Computing Machinery.

Fengzhu Zeng and Wei Gao. 2024. JustiLM: Few-shot justification generation for explainable fact-checking of real-world claims. Transactions of the Association for Computational Linguistics, 12:334–354.

Yirong Zeng, Xiao Ding, Yi Zhao, Xiangyu Li, Jie Zhang, Chao Yao, Ting Liu, and Bing Qin. 2024. RU22Fact: Optimizing evidence for multilingual explainable fact-checking on Russia-Ukraine conflict. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 14215–14226, Torino, Italia. ELRA and ICCL.
Appendix

A  Dataset Creation

A.1  Claim Selection

Table 6 lists the reliable publisher websites used for collecting EuroVerdict’s claims and their related fact-checking articles for each language.

A.2  Annotation Guidelines

Hereafter we provide the annotation guidelines:

Please open the dataset you have been provided with according to the language you are working with and follow these instructions.

1. The first goal is to reduce the database to at least 200 claims. For this, first apply a filter on Rating and select only: false, partly false and missing context (or equivalent in the language you are working with).

2. Please keep in mind that the fact-checkers providing content to the Google Fact Check Explorer have different ways of naming the ratings but you should find those referring to false, partly false and missing context.

3. Then, discard those claims that are Political fact-checks, meaning those claims attributed directly to what a politician said. You can do this by erasing those claims of political fact-checks directly in the database. For example, these of Kamala Harris and Donald Trump:

4. If this did not reduce the database to at least 200 claims, please let us know through email.

5. Once you have a dataset of between 100-200 claims you can proceed to fill in the fields of Verdict and Relevant Links.

- Verdict: a few sentences long text explaining why the claim is false/partly false, etc. This works as evidence or as the basis for the rating. It should be as neutral as possible. For example, in this claim: “The Pope Francis has been seen partying and drinking alcohol”, the verdict would be: “Details in the images (that were circulated online) that may indicate that they have been created with AI tools have been detected. Also it was proved that the social media profiles disseminating the content were defined as ‘meme accounts’”. When discussing images or video the verdict should be comprehensible even without viewing the media as in the provided example.

- Relevant Links: provide a list of links on which the verdict is based (i.e. containing relevant evidence, they can be information outside of your organization). The relevant evidence should be text based (i.e. no direct link to video or images). -

6. Please let us know whenever you complete the task or if you have any questions.

B  Dataset Analysis

B.1  Topic Modeling

For topic modeling, we implemented the BERTopic strategy using BAAI/bge-m3 as the embedding model. Dimensionality reduction was performed with UMAP (n_neighbors=15, n_components=20, min_dist=0.0, metric="cosine", random_state=42), while clustering was carried out using HDBSCAN min_cluster_size=10, metric="euclidean", cluster_selection_method="eom", prediction_data=True).

To automatically assign cluster-specific labels, we employed Llama-3.1-8B-Instruct for one-shot classification, using the following prompt.12

SYSTEM: You are a helpful, respectful and honest assistant for labeling topics.
EXAMPLE PROMPT: I have a topic that contains the following multilingual documents: - Traditional diets in most cultures were primarily plant-based with a little meat on top, but with the rise of industrial style meat production and factory farming, meat has become a staple food.
- Meat, but especially beef, is the word food in terms of emissions.
- Eating meat doesn’t make you a bad person, not eating meat doesn’t make you a good one.
The topic is described by the following multilingual keywords: 'meat, beef, eat, eating, emissions, steak, food, health, processed, chicken'.
Based on the information about the topic above, please create a short English label for this topic. Make sure you only return the label and nothing more. ***
EXAMPLE PROMPT REPLY: Environmental impacts of eating meat
MAIN PROMPT: I have a topic that contains the following multilingual documents: [DOCUMENTS]
The topic is described by the following multilingual keywords: '[KEYWORDS]'.
Based on the information about the topic above, please create a short English label for this topic. Make sure you only return the label and nothing more.

For visualization, we further reduced the embeddings to two components using UMAP

12We adapted the prompt provided on the BERTopic documentation https://maartengr.github.io/BERTopic/getting_started/representation/llm.html#llama-2
<table>
  <tr>
    <th>Language</th>
    <th>Publishers</th>
  </tr>
  <tr>
    <td>RO</td>
    <td>verificat.afp.com, brodhub.eu, factual.ro</td>
  </tr>
  <tr>
    <td>EL</td>
    <td>ellinikaoaxes.gr, factcheckgreek.afp.com, factreview.gr</td>
  </tr>
  <tr>
    <td>IT</td>
    <td>facta.news, bufale.net, pagellapolitica.it</td>
  </tr>
  <tr>
    <td>EN</td>
    <td>cjp.org.in, logicallyfacts.com, newschecker.in, rappler.com, thequint.com, factcheck.afp.com, actcheck.org, checkyourfact.com, thip.media, snopes.com, newsweek.com, politifact.com, polygraph.info, fullfact.org</td>
  </tr>
  <tr>
    <td>FR</td>
    <td>francetinfo.fr, factuel.afp.com, dpa-factchecking.com, guineechek.org, tf1info.fr, observers.france24.com, francetinfo.fr, defacto-observatoire.fr</td>
  </tr>
  <tr>
    <td>PL</td>
    <td>demagog.org.pl, sprawdzam.afp.com, oko.press</td>
  </tr>
  <tr>
    <td>ES</td>
    <td>verifica.efe.com, fastcheck.cl, newtral.es, factual.afp.com</td>
  </tr>
  <tr>
    <td>DE</td>
    <td>dpa-factchecking.com, correctiv.org, faktenchec.k.afp.com, br.de</td>
  </tr>
</table>

Table 6: reliable publisher websites

n_neighbors=15, n_components=2, min_dist=0.0, metric="cosine", random_state=42). In Figure 2 we present the topic for all EuroVerdict data. Language-specific topic modeling visualisation are presented in Figure 4.

C Experimental Design Details

C.1 Dataset Partition

For the experiments, we divided EuroVerdict into three subsets: train, evaluation, and test sets. In Table 7, we report the number of items in each set, both for the entire dataset and for each individual language.

<table>
  <tr>
    <th></th>
    <th>Train</th>
    <th>Eval</th>
    <th>Test</th>
  </tr>
  <tr>
    <td>All</td>
    <td>1152</td>
    <td>245</td>
    <td>245</td>
  </tr>
  <tr>
    <td>DE</td>
    <td>142</td>
    <td>31</td>
    <td>31</td>
  </tr>
  <tr>
    <td>EL</td>
    <td>138</td>
    <td>30</td>
    <td>30</td>
  </tr>
  <tr>
    <td>EN</td>
    <td>138</td>
    <td>30</td>
    <td>30</td>
  </tr>
  <tr>
    <td>ES</td>
    <td>186</td>
    <td>40</td>
    <td>40</td>
  </tr>
  <tr>
    <td>FR</td>
    <td>135</td>
    <td>29</td>
    <td>29</td>
  </tr>
  <tr>
    <td>IT</td>
    <td>143</td>
    <td>31</td>
    <td>31</td>
  </tr>
  <tr>
    <td>PL</td>
    <td>143</td>
    <td>32</td>
    <td>32</td>
  </tr>
  <tr>
    <td>RO</td>
    <td>135</td>
    <td>30</td>
    <td>30</td>
  </tr>
</table>

Table 7: Data distribution across train, eval, and test sets.

C.2 Expert-Based LLM selection

To assess whether Llama-31-8B-Instruct could not only generate grammatically correct text in all eight languages of EuroVerdict but also produce plausible verdicts, we conducted a preliminary zero-shot verdict generation experiment followed by a human evaluation. Similar to the main experiments, we provided the model with a subset of claims in all eight languages along with their corresponding fact-checking articles and instructed it to generate verdicts based on the provided information. The generated verdicts were then evaluated by expert annotators, who were native speakers of the respective languages, and rated on a scale from 1 to 5 for grammatical correctness and soundness (i.e., how closely the generated verdict resembles a real verdict). Results showed that across all languages, the generated verdicts were considered both grammatically correct and plausible. Based on these findings, we proceeded to employ the LLM throughout the experiments presented in this work.

C.3 Chain-of-Thought Prompt

In the following, we provide the prompt employed in the Chain-of-Thought configuration.

SYSTEM: You will be provided with a misleading claim. Evaluate this claim step-by-step using the given article. To do so, answer to the following questions (reasoning). Finally state whether the claim is true or false (veracity label), and provide a short reply to the claim providing the reasons why the claim is true or false (justification).
1. What does the claim specifically state?
2. What context or background information in the article is relevant to evaluating the claim?
3. What evidence supports or contradicts the claim?
4. Are there exceptions or scenarios where the claim doesn’t apply?
5. To what extent is the claim accurate or misleading? Provide a conclusion with key caveats or nuances.
USER: <claim> "{claim}" <claim>
<article>"{article}" <article>
Reply in {language}. Return the response in JSON format: {"claim": "claim", "reasoning": [list of the replies to each question], "veracity_label": "TRUE or FALSE", "justification": "why the claim is true or false"} Leave the JSON names in English. Ensure the structure remains consistent throughout.

C.4 Fine-Tuning Details

We utilized Llama-3.1-8B-Instruct for verdict generation. Four versions of the LMM were fine-
(a) German                (b) Greek                (c) English
(d) Spanish               (e) French               (f) Italian
(g) Polish                (h) Romanian

Figure 4: Topic analysis for each language in EuroVerdict dataset.

tuned, combining English and language specific prompts with the original fact-checking articles and the English translated ones. The same hyperparameters were used across all the fine-tuning, with the only variation being the training data. In particular, we employed the prompt presented in Paragraph 4, adding the gold verdict as for the assistant role. The training was performed on an NVIDIA Ampere A40 GPU with 48GB of memory, applying a 4bit quantization. Low-Rank Adaptation (LoRA) was utilized with a rank of 16, an \( \alpha \) value of 16, and a dropout rate of 0. Training parameters included a learning rate of \( 5 \times 10^{-5} \), a training and evaluation batch size of 6, and gradient accumulation steps of 4. The model was trained for 5 epochs, with a weight decay of 0.01, and a warm-up ratio of 0.03. For inference, we employed the checkpoint with the lowest evaluation loss.
<table>
  <tr>
    <th></th>
    <th>Overall Mean</th>
    <th>Alignment</th>
    <th>Understandability</th>
    <th>Soundness</th>
  </tr>
  <tr>
    <th rowspan="4">Fine-Tuning</th>
    <td>OA-EN</td>
    <td>4.255</td>
    <td>4.118</td>
    <td>4.412</td>
    <td>4.235</td>
  </tr>
  <tr>
    <td>OA-lang</td>
    <td>4.235</td>
    <td>4.118</td>
    <td>4.471</td>
    <td>4.118</td>
  </tr>
  <tr>
    <td>TA-EN</td>
    <td>3.159</td>
    <td>3.13</td>
    <td>3.391</td>
    <td>2.957</td>
  </tr>
  <tr>
    <td>TA-lang</td>
    <td>3.377</td>
    <td>3.435</td>
    <td>3.565</td>
    <td>3.13</td>
  </tr>
  <tr>
    <th rowspan="4">Zero-shot</th>
    <td>OA-EN</td>
    <td>4.104</td>
    <td>4.188</td>
    <td>4.312</td>
    <td>3.812</td>
  </tr>
  <tr>
    <td>OA-lang</td>
    <td>3.863</td>
    <td>3.647</td>
    <td>4.412</td>
    <td>3.529</td>
  </tr>
  <tr>
    <td>TA-EN</td>
    <td>3.273</td>
    <td>3.409</td>
    <td>3.227</td>
    <td>3.182</td>
  </tr>
  <tr>
    <td>TA-lang</td>
    <td>3.364</td>
    <td>3.500</td>
    <td>3.500</td>
    <td>3.091</td>
  </tr>
  <tr>
    <th rowspan="2">CoT</th>
    <td>OA-EN</td>
    <td>3.725</td>
    <td>3.824</td>
    <td>3.588</td>
    <td>3.765</td>
  </tr>
  <tr>
    <td>TA-EN</td>
    <td>2.681</td>
    <td>3.000</td>
    <td>2.435</td>
    <td>2.609</td>
  </tr>
</table>

Table 8: Human evaluation results averaged across all languages. We report the overall score (Overall Mean) and scores for Alignment, Understandability, and Soundness across configurations: fine-tuning, zero-shot, CoT, original (OA) or translated articles (TA), and English (EN) or language-specific prompts (lang).

D Experimental Results Details

D.1 Verdict Generation Results

In Tables In Tables 9 and 10, we present the results for all the experiments. For clarity, experiments with articles in the original language are reported in Table 9, while those with articles automatically translated into English are shown in Table 10.

D.2 Human Evaluation Instructions

Hereafter, we report the task proposed to the human evaluators.

We prepared a list of Claims and Gold Verdicts.

For each of them we provide a list of 5 additional verdicts, that we ask you to evaluate according to:

• Alignment: How much is the current verdict similar to the gold verdict?
• Grammaticality: How much is this verdict written in an understandable way (regardless of being a proper verdict)?
• Soundness: How much does the verdict sounds like a well formed verdict?

Score each of the provided verdicts on a scale from 1 to 5, according to the following guidelines:

Alignment

5. Arguments and conclusions are semantically the same
4. Arguments and conclusions are very similar
3. Arguments and conclusions have some discrepancies
2. The conclusion is similar but arguments are completely different (or vice versa)
1. Arguments and conclusions are completely different

Grammaticality

5. Understandable and grammatical
4. Understandable but with few grammatical errors
3. Understandable but with several grammatical errors
2. Difficult to understand with severe grammatical errors
1. Not understandable"

Soundness

5. Very convincing verdict
4. Proper verdict
3. Acceptable Verdict
2. Not a proper verdict
1. Does not seem like a legit verdict"

D.3 Human Evaluation Results Details

In Table 8 we report the scores obtained from the human evaluation of the generated verdicts. Details are provided in Section 5.1.
<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="4">ROUGE-1</th>
    <th colspan="4">ROUGE-L</th>
    <th colspan="4">BertScore</th>
    <th colspan="4">Cosine Similarity</th>
  </tr>
  <tr>
    <th>EL</th><th>LS</th><th>EL</th><th>LS</th>
    <th>EL</th><th>LS</th><th>EL</th><th>LS</th>
    <th>EL</th><th>LS</th><th>EL</th><th>LS</th>
    <th>EL</th><th>LS</th>
  </tr>
  <tr>
    <td rowspan="3">DE</td>
    <td>Zero-shot</td><td>0.293</td><td>0.288</td><td>0.243</td><td>0.248</td><td>0.752</td><td>0.750</td><td>0.650</td><td>0.654</td>
    <td>Fine-tune</td><td><b>0.378</b></td><td><b>0.393</b></td><td><b>0.353</b></td><td><b>0.362</b></td><td><b>0.786</b></td><td><b>0.792</b></td><td><b>0.704</b></td><td><b>0.724</b></td>
    <td>CoT</td><td>0.221</td><td>-</td><td>0.176</td><td>-</td><td>0.713</td><td>-</td><td>0.652</td><td>-</td>
  </tr>
  <tr>
    <td>Zero-shot</td><td>0.294</td><td>0.347</td><td>0.217</td><td>0.257</td><td>0.721</td><td>0.729</td><td>0.599</td><td>0.657</td>
    <td>Fine-tune</td><td><b>0.562</b></td><td><b>0.662</b></td><td><b>0.507</b></td><td><b>0.628</b></td><td><b>0.821</b></td><td><b>0.860</b></td><td><b>0.778</b></td><td><b>0.820</b></td>
    <td>CoT</td><td>0.267</td><td>-</td><td>0.180</td><td>-</td><td>0.694</td><td>-</td><td>0.626</td><td>-</td>
  </tr>
  <tr>
    <td rowspan="3">EN</td>
    <td>Zero-shot</td><td>0.429</td><td>0.419</td><td>0.305</td><td>0.287</td><td>0.769</td><td>0.758</td><td><b>0.760</b></td><td><b>0.739</b></td>
    <td>Fine-tune</td><td><b>0.470</b></td><td><b>0.494</b></td><td><b>0.373</b></td><td><b>0.383</b></td><td><b>0.784</b></td><td><b>0.786</b></td><td>0.723</td><td>0.733</td>
    <td>CoT</td><td>0.391</td><td>-</td><td>0.263</td><td>-</td><td>0.741</td><td>-</td><td>0.730</td><td>-</td>
  </tr>
  <tr>
    <td>Zero-shot</td><td>0.388</td><td>0.340</td><td>0.301</td><td>0.257</td><td>0.758</td><td>0.743</td><td>0.633</td><td>0.580</td>
    <td>Fine-tune</td><td><b>0.474</b></td><td><b>0.438</b></td><td><b>0.406</b></td><td><b>0.346</b></td><td><b>0.799</b></td><td><b>0.779</b></td><td><b>0.716</b></td><td><b>0.696</b></td>
    <td>CoT</td><td>0.356</td><td>-</td><td>0.269</td><td>-</td><td>0.747</td><td>-</td><td>0.666</td><td>-</td>
  </tr>
  <tr>
    <td rowspan="3">FR</td>
    <td>Zero-shot</td><td>0.307</td><td>0.290</td><td>0.209</td><td><b>0.193</b></td><td>0.724</td><td><b>0.717</b></td><td><b>0.648</b></td><td><b>0.645</b></td>
    <td>Fine-tune</td><td><b>0.327</b></td><td><b>0.299</b></td><td><b>0.220</b></td><td><b>0.193</b></td><td><b>0.726</b></td><td>0.714</td><td>0.616</td><td>0.601</td>
    <td>CoT</td><td>0.273</td><td>-</td><td>0.169</td><td>-</td><td>0.706</td><td>-</td><td>0.611</td><td>-</td>
  </tr>
  <tr>
    <td>Zero-shot</td><td>0.288</td><td>0.273</td><td>0.185</td><td><b>0.191</b></td><td>0.722</td><td>0.713</td><td><b>0.694</b></td><td>0.607</td>
    <td>Fine-tune</td><td><b>0.293</b></td><td><b>0.298</b></td><td><b>0.201</b></td><td><b>0.189</b></td><td><b>0.731</b></td><td><b>0.723</b></td><td>0.671</td><td><b>0.674</b></td>
    <td>CoT</td><td>0.250</td><td>-</td><td>0.166</td><td>-</td><td>0.708</td><td>-</td><td>0.611</td><td>-</td>
  </tr>
  <tr>
    <td rowspan="3">PL</td>
    <td>Zero-shot</td><td>0.291</td><td>0.238</td><td><b>0.228</b></td><td>0.176</td><td>0.706</td><td>0.690</td><td><b>0.696</b></td><td>0.609</td>
    <td>Fine-tune</td><td><b>0.278</b></td><td><b>0.267</b></td><td><b>0.210</b></td><td><b>0.214</b></td><td><b>0.709</b></td><td><b>0.717</b></td><td>0.692</td><td><b>0.725</b></td>
    <td>CoT</td><td>0.176</td><td>-</td><td>0.133</td><td>-</td><td>0.674</td><td>-</td><td>0.596</td><td>-</td>
  </tr>
  <tr>
    <td>Zero-shot</td><td>0.290</td><td>0.292</td><td>0.198</td><td><b>0.213</b></td><td>0.710</td><td>0.712</td><td>0.640</td><td>0.670</td>
    <td>Fine-tune</td><td><b>0.292</b></td><td>0.285</td><td><b>0.201</b></td><td>0.200</td><td><b>0.712</b></td><td>0.712</td><td><b>0.675</b></td><td><b>0.681</b></td>
    <td>CoT</td><td>0.202</td><td>-</td><td>0.139</td><td>-</td><td>0.685</td><td>-</td><td>0.657</td><td>-</td>
  </tr>
  <tr>
    <td rowspan="3">RO</td>
    <td>Zero-shot</td><td>0.325</td><td>0.311</td><td>0.238</td><td>0.229</td><td>0.734</td><td>0.727</td><td>0.669</td><td>0.642</td>
    <td>Fine-tune</td><td><b>0.387</b></td><td><b>0.393</b></td><td><b>0.312</b></td><td><b>0.315</b></td><td><b>0.760</b></td><td><b>0.761</b></td><td><b>0.698</b></td><td><b>0.707</b></td>
    <td>CoT</td><td>0.270</td><td>-</td><td>0.190</td><td>-</td><td>0.710</td><td>-</td><td>0.644</td><td>-</td>
  </tr>
  <tr>
    <td colspan="4"><i>Mean</i></td>
    <td colspan="4"></td>
    <td colspan="4"></td>
    <td colspan="4"></td>
  </tr>
</table>

Table 9: Experimental results of Llama-3.1-8B-Instruct on the EuroVerdict dataset. We report performance across Zero-Shot, Fine-Tuning (using articles in the original language), and Chain-of-Thought (CoT) settings. Evaluation metrics include ROUGE-1, ROUGE-L, BERTScore, and Cosine Similarity. For each language, results are presented for two prompt types: EL (English-language prompts) and LS (language-specific prompts).
<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="4">ROUGE-1</th>
    <th colspan="4">ROUGE-L</th>
    <th colspan="4">BertScore</th>
    <th colspan="4">Cosine Similarity</th>
  </tr>
  <tr>
    <th>EL</th><th>LS</th><th>EL</th><th>LS</th>
    <th>EL</th><th>LS</th><th>EL</th><th>LS</th>
    <th>EL</th><th>LS</th><th>EL</th><th>LS</th>
    <th>EL</th><th>LS</th>
  </tr>
  <tr>
    <td rowspan="3">DE</td>
    <td>Zero-shot</td><td>0.264</td><td>0.281</td><td>0.211</td><td>0.240</td>
    <td>0.732</td><td>0.745</td><td>0.641</td><td>0.670</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td>
  </tr>
  <tr>
    <td>Fine-tune</td><td><b>0.422</b></td><td><b>0.351</b></td><td><b>0.381</b></td><td><b>0.314</b></td>
    <td><b>0.805</b></td><td><b>0.777</b></td><td><b>0.722</b></td><td><b>0.681</b></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td>
  </tr>
  <tr>
    <td>CoT</td><td>0.200</td><td>-</td><td>0.159</td><td>-</td>
    <td>0.701</td><td>-</td><td>0.634</td><td>-</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td>
  </tr>
  <tr>
    <td rowspan="3">EL</td>
    <td>Zero-shot</td><td>0.265</td><td>0.296</td><td>0.190</td><td>0.196</td>
    <td>0.698</td><td>0.701</td><td>0.608</td><td>0.647</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td>
  </tr>
  <tr>
    <td>Fine-tune</td><td><b>0.343</b></td><td><b>0.349</b></td><td><b>0.280</b></td><td><b>0.284</b></td>
    <td><b>0.731</b></td><td><b>0.730</b></td><td><b>0.695</b></td><td><b>0.744</b></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td>
  </tr>
  <tr>
    <td>CoT</td><td>0.226</td><td>-</td><td>0.162</td><td>-</td>
    <td>0.675</td><td>-</td><td>0.566</td><td>-</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td>
  </tr>
  <tr>
    <td rowspan="3">EN</td>
    <td>Zero-shot</td><td>0.437</td><td>0.422</td><td>0.298</td><td>0.299</td>
    <td>0.766</td><td>0.766</td><td>0.718</td><td>0.742</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td>
  </tr>
  <tr>
    <td>Fine-tune</td><td><b>0.472</b></td><td><b>0.515</b></td><td><b>0.367</b></td><td><b>0.407</b></td>
    <td><b>0.782</b></td><td><b>0.798</b></td><td><b>0.737</b></td><td><b>0.754</b></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td>
  </tr>
  <tr>
    <td>CoT</td><td>0.381</td><td>-</td><td>0.255</td><td>-</td>
    <td>0.739</td><td>-</td><td>0.710</td><td>-</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td>
  </tr>
  <tr>
    <td rowspan="3">ES</td>
    <td>Zero-shot</td><td>0.345</td><td>0.340</td><td>0.256</td><td>0.257</td>
    <td>0.751</td><td>0.743</td><td>0.675</td><td>0.645</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td>
  </tr>
  <tr>
    <td>Fine-tune</td><td><b>0.395</b></td><td><b>0.405</b></td><td><b>0.294</b></td><td><b>0.329</b></td>
    <td><b>0.770</b></td><td><b>0.770</b></td><td><b>0.869</b></td><td><b>0.676</b></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td>
  </tr>
  <tr>
    <td>CoT</td><td>0.316</td><td>-</td><td>0.232</td><td>-</td>
    <td>0.734</td><td>-</td><td>0.603</td><td>-</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td>
  </tr>
  <tr>
    <td rowspan="3">FR</td>
    <td>Zero-shot</td><td><b>0.303</b></td><td>0.270</td><td><b>0.213</b></td><td>0.182</td>
    <td><b>0.719</b></td><td>0.710</td><td><b>0.632</b></td><td>0.614</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td>
  </tr>
  <tr>
    <td>Fine-tune</td><td>0.275</td><td><b>0.307</b></td><td>0.181</td><td><b>0.195</b></td>
    <td>0.712</td><td><b>0.718</b></td><td>0.605</td><td><b>0.637</b></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td>
  </tr>
  <tr>
    <td>CoT</td><td>0.262</td><td>-</td><td>0.159</td><td>-</td>
    <td>0.697</td><td>-</td><td>0.601</td><td>-</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td>
  </tr>
  <tr>
    <td rowspan="3">IT</td>
    <td>Zero-shot</td><td>0.261</td><td>0.289</td><td>0.177</td><td>0.202</td>
    <td>0.710</td><td>0.721</td><td>0.652</td><td>0.652</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td>
  </tr>
  <tr>
    <td>Fine-tune</td><td><b>0.277</b></td><td><b>0.293</b></td><td><b>0.196</b></td><td><b>0.207</b></td>
    <td><b>0.717</b></td><td><b>0.725</b></td><td><b>0.666</b></td><td><b>0.662</b></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td>
  </tr>
  <tr>
    <td>CoT</td><td>0.233</td><td>-</td><td>0.158</td><td>-</td>
    <td>0.700</td><td>-</td><td>0.606</td><td>-</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td>
  </tr>
  <tr>
    <td rowspan="3">PL</td>
    <td>Zero-shot</td><td><b>0.270</b></td><td>0.209</td><td><b>0.205</b></td><td>0.156</td>
    <td><b>0.705</b></td><td>0.674</td><td><b>0.732</b></td><td>0.637</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td>
  </tr>
  <tr>
    <td>Fine-tune</td><td>0.232</td><td><b>0.232</b></td><td>0.180</td><td><b>0.178</b></td>
    <td>0.701</td><td><b>0.697</b></td><td>0.703</td><td><b>0.660</b></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td>
  </tr>
  <tr>
    <td>CoT</td><td>0.207</td><td>-</td><td>0.171</td><td>-</td>
    <td>0.672</td><td>-</td><td>0.589</td><td>-</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td>
  </tr>
  <tr>
    <td rowspan="3">RO</td>
    <td>Zero-shot</td><td><b>0.264</b></td><td>0.255</td><td>0.180</td><td><b>0.181</b></td>
    <td><b>0.706</b></td><td>0.699</td><td><b>0.641</b></td><td><b>0.652</b></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td>
  </tr>
  <tr>
    <td>Fine-tune</td><td>0.260</td><td><b>0.272</b></td><td><b>0.195</b></td><td>0.178</td>
    <td>0.699</td><td><b>0.703</b></td><td>0.636</td><td>0.628</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td>
  </tr>
  <tr>
    <td>CoT</td><td>0.226</td><td>-</td><td>0.152</td><td>-</td>
    <td>0.686</td><td>-</td><td>0.621</td><td>-</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td>
  </tr>
  <tr>
    <td rowspan="3"><i>Mean</i></td>
    <td>Zero-shot</td><td>0.302</td><td>0.296</td><td>0.218</td><td>0.216</td>
    <td>0.724</td><td>0.721</td><td>0.664</td><td>0.657</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td>
  </tr>
  <tr>
    <td>Fine-tune</td><td><b>0.336</b></td><td><b>0.342</b></td><td><b>0.261</b></td><td><b>0.264</b></td>
    <td><b>0.741</b></td><td><b>0.741</b></td><td><b>0.682</b></td><td><b>0.680</b></td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td>
  </tr>
  <tr>
    <td>CoT</td><td>0.258</td><td>-</td><td>0.183</td><td>-</td>
    <td>0.702</td><td>-</td><td>0.616</td><td>-</td>
    <td></td><td></td><td></td><td></td>
    <td></td><td></td>
  </tr>
</table>

Table 10: Experimental results of Llama-3.1-8B-Instruct on the EuroVerdict dataset. We report performance across Zero-Shot, Fine-Tuning (using articles translated in English), and Chain-of-Thought (CoT) settings. Evaluation metrics include ROUGE-1, ROUGE-L, BERTScore, and Cosine Similarity. For each language, results are presented for two prompt types: EL (English-language prompts) and LS (language-specific prompts).

Reviewer paper summaries: This paper introduces EuroVerdict, a novel multilingual dataset explicitly designed for generating explanatory verdicts in automated fact-checking. The dataset covers eight European languages and was developed in collaboration with professional fact-checkers, who manually curated claims, verdicts, and supporting evidence. The authors evaluated the dataset using the Llama-3.1-8B-Instruct model under various multilingual prompting strategies, including zero-shot, Chain-of-Thought (CoT), and fine-tuning methods.

The experiments show better results of fine-tuning approaches over zero-shot and CoT, particularly in multilingual contexts. The authors test performance through generation evaluation metrics (ROUGE, BERTScore, cosine similarity) and human evaluations conducted by professional fact-checkers.

Summary: This paper collected a multilingual dataset EuroVerdict designed for verdict generation, covering eight European languages, with 200 instances for each, resulting in a total of 1642 cases. Each entry includes a claim, a manually written verdict, evidence articles and additional evidence source links. Based on EuroVerdict, they evaluated llama 3.1-8B-instruct for verdict generation based on the claim and the corresponding evidence under three settings. Based on their evaluation criteria which prioritizes alignment with human-written verdicts, experimental results show that CoT performs the worst.
Summary of strengths: - Multilingual dataset specifically tailored to explanatory verdict generation in automated fact-checking, covering eight European languages. This makes it a valuable foundational resource for future multilingual studies.
- The dataset was curated through collaboration with professional fact-checkers, enhancing its reliability, practical relevance, and authenticity.

EuroVerdict would be a useful multilingual dataset for evaluating automatic fact-checking systems and assess models’ capability in generating verdicts.
Summary of weaknesses: - The authors don’t provide information regarding inter-annotator agreement, despite annotation being central to the dataset’s validity. This would help quantify annotation reliability.  Given the context and dataset’s limited size (~200 entries per language), potential biases or variability in annotation quality remain unexplored.
- The dataset test set sizes are limited (~30-40 examples per language), potentially introducing significant evaluation bias. 
- Although the authors briefly mention (lines 466–484) several possible reasons for the Chain-of-Thought (CoT) strategy’s poor performance (e.g., error propagation, language mismatch), these hypotheses are presented without any evidence or systematic error analysis.
- Need for a deeper investigation into why CoT prompts perform worse, such as examining step-by-step generated reasoning or intermediate outputs. The study exclusively uses metrics designed for generation tasks (e.g., ROUGE, BERTScore, cosine similarity) without evaluating label accuracy (e.g., precision, recall, F1-score) for the claim verification decisions, at least for CoT model that was required to produce also a Label. Given that they have explicit labels from professional annotators, the authors indeed could have included standard classification accuracy metrics such as accuracy, precision, recall, and F1-score, alongside their existing generation-focused metrics. 
- The authors rely on Google Translate to evaluate multilinguality without providing an analysis of translation accuracy or potential mistranslation errors. Given the sensitivity of fact-checking tasks to nuanced language details (e.g., negations, conditional statements), even minor translation errors could substantially affect model performance and interpretation of results.
- The authors refer to “a pool of professional fact-checkers,” (Section 5.1) but don’t specify the exact number of evaluators, their expertise beyond being native speakers, and the sampling strategy for assigning tasks and the inter-annotator agreement.

- **Identify the goal of building up this dataset and more comprehensive experiments may be needed**. In my view, it can serve as a benchmark (1) evaluating the effectiveness of automatic fact-checking systems, referring to verdict prediction in this paper, and (2) examining LLMs’ capability of generating verdicts. Of course, there would be other verdict generation methods which are not based on LLMs. This work evaluated the second but miss the first. For the second, more models can be evaluated, particularly some representative model families, such as GPT, Claude, Gemini. For llama, a larger size can be tried.

- **The evaluation philosophy seems not that reasonable**, and the evaluation metrics by regarding human-written verdicts as gold and contrasting with LLM-generated verdicts may be not comprehensive, and then the findings based on these evaluation numbers may not reflect the true situations.
There are three criteria described in lines 504-509: alignment with human-written verdicts, understandability and soundness, which one is the priority? The evaluation metrics of ROUGE and BERTScore and semantic similarity all emphasize the alignment. If authors would like to consider all and assign different weights, LLM-as-a-judge can be considered, but here llama3.1-8B would not a good judge, better use some powerful models like GPT-4o.

- **Evaluations only based on llama 3.1-8B-instruct may bias the results and findings**. llama-8B is a small open-source model, it performs poorly on reasoning. So in my opinion, it is not because CoT approach in verdict generation is the worst, but two reasons make it the worst: (1) llama3.1-8B is not capable enough to be effective in reasoning, and (2) evaluation metrics prioritizing alignment with human verdicts do not favor generations stemming from reasoning process. Since after CoT, the generation tends to be long and detailed, 3 sentences could not empower the function of CoT. Given that the current evaluation metrics favor well-aligned generations, I guess few-shot would get good results. We even do not need fine-tuning, especially for models like GPT-4o.
Comments, suggestions and typos: L218 - English repeated

**Suggestions:**
- authors can clarify what the article for each entry refers to at the beginning clearly, because it can be understood as the source article from which the claim was extracted, while it can also be regarded as the evidence article supporting the verdicts. In my understanding, it is the evidence.
- Line 102: To evaluate EuroVerdict → Based on EuroVerdict, otherwise to evaluate what aspects of EuroVerdict?
- Line 108-131: many sentences express the same meaning, deduplicate and make it concisely
- Tables and Figures should be on top by [t!], see ACL writing guideline 
- Figure 3, legend, soundess -> soundness

**Question:**
- which part of data is used for fine-tuning, full set or some splits? What loss function is used for fine-tuning, similar to instruction tuning or different?
Overall assessment scores: 2

2