Venue: EMNLP 2023

# Multilingual Previously Fact-Checked Claim Retrieval

 Matus Pikuliak, Ivan Srba, Robert Moro, Timo Hromadka, Timotej Smolen Martin Melisek, Ivan Vykopal, Jakub Simko, Juraj Podrouzek, Maria Bielikova

Kempelen Institute of Intelligent Technologies

{name.surname}@kinit.sk

###### Abstract

Fact-checkers are often hampered by the sheer amount of online content that needs to be fact-checked. NLP can help them by retrieving already existing fact-checks relevant to the content being investigated. This paper introduces a new multilingual dataset - _MultiClaim_ - for previously fact-checked claim retrieval. We collected 28k posts in 27 languages from social media, 206k fact-checks in 39 languages written by professional fact-checkers, as well as 31k connections between these two groups. This is the most extensive and the most linguistically diverse dataset of this kind to date. We evaluated how different unsupervised methods fare on this dataset and its various dimensions. We show that evaluating such a diverse dataset has its complexities and proper care needs to be taken before interpreting the results. We also evaluated a supervised fine-tuning approach, improving upon the unsupervised method significantly.

+
Footnote †: The dataset is available at Zenodo upon request _for research purposes only_: [https://zenodo.org/record/7737983](https://zenodo.org/record/7737983). The source code is available at: [https://github.com/kinin-sk/multiclaim](https://github.com/kinin-sk/multiclaim).

## 1 Introduction

Fact-checking organizations have made progress in recent years in manually and professionally fact-checking viral content [45, 4]. To reduce some of the fact-checkers' manual efforts and make their work more effective, several studies have recently examined their needs and identified tasks that could be automated [23, 24, 25, 26, 27]. These tasks include searching for the source of evidence for verification, searching for other versions of misinformation, and searching within existing fact-checks. They were identified as particularly challenging for fact-checkers working in low-resource languages [11].

In this work, we focus on _previously fact-checked claim retrieval_ (PFCR) [28]. Given a text making an _input claim_ (e.g., a social media post) and a set of _fact-checked claims_, our task is to rank the _fact-checked claims_ so that those that are the most relevant w.r.t. the _input claim_ (and thus the most useful from the fact-checker's perspective) are ranked as high as possible.

Previously, this task was mostly done in English. Other languages that have been considered include Arabic [23], Bengali, Hindi, Malayam, and Tamil [26]. However, many other languages or even entire major language families have not been considered at all. Additionally, so far only _monolingual PFCR_ has been tackled, when the input claim and the fact-checked claims are in the same language. To address these shortcomings, we introduce in this paper a new extensive multilingual dataset. Our two main contributions are:

* Multilingual dataset for PFCR.We collected and made available1 a novel multilingual dataset for PFCR.The dataset consists of 205,751 fact-checks in 39 languages and 28,092 social media posts (from now on just _posts_) in 27 languages. For most of these languages, this is the first time this task has been considered at all. This is also the biggest dataset of fact-checks released to date.

Footnote 1: The dataset is available at Zenodo upon request _for research purposes only_: [https://zenodo.org/record/7737983](https://zenodo.org/record/7737983). The source code is available at: [https://github.com/kinin-sk/multiclaim](https://github.com/kinin-sk/multiclaim).

All the posts were previously reviewed by professional fact-checkers who also assigned appropriate fact-checks to them. We collected these assignments and gathered 31,305 pairs consisting of a post and a fact-check reviewing the claim made in the post. 4,212 of these pairs are crosslingual (i.e., the language of the fact-check and the language of the post are different). This dataset introduces _crosslingual PFCR_ as a new task that has not been tackled before. This is the biggest collection of such pairs that were confirmed by professional fact-checkers.



OCR transcripts of the images attached to the posts and machine translation of all the data into English.

2. In-depth multilingual evaluation.We evaluated the performance of various text embedding models and BM25 for both the original multilingual data and their English translations. We describe several pitfalls related to the complexity of evaluating such a linguistically diverse dataset. We also explore the performance across several other data dimensions, such as post length or publication date. Finally, we show that we can improve text embedding methods further by using supervised training with our data.

## 2 Related Work

Other names are used for PFCR or similar tasks for various reasons, e.g., fact-checking URL recommendation Vo and Lee (2018), fact-checked claims detection Shaar et al. (2020), verified claim retrieval Barron-Ccedeno et al. (2020), searching for fact-checked information Vo and Lee (2020), or claim matching Kazemi et al. (2021).

Datasets._CheckThat!_ datasets Barron-Ccedeno et al. (2020); Shaar et al. (2021) have the most similar collection approach to ours. They collect English and Arabic tweets mentioned in fact-checks to create preliminary pairs and then manually filter them. Compared to this work, we broaden the scope of data collection and omit the manual cleaning in favor of using fact-checkers' reports. Shaar et al. (2020) collected data from fact-checking of English political debates done by fact-checkers. The _CrowdChecked_ dataset Hardalov et al. (2022) was created by searching for fact-check URLs on Twitter and collecting English tweets from retrieved threads. The process is inherently noisy and, the authors propose different noise filtering techniques.

Kazemi et al. (2021) collected several million chat messages from public chat groups and tiplines in English, Bengali, Hindi, Malayalam, and Tamil and 150k fact-checks. Then they sampled roughly 2,300 pairs based on their embedding similarity and manually annotated them. In the end, they obtained only roughly 250 positive pairs. Jiang et al. (2021) matched COVID-19 tweets and 90 COVID-19 claims in a similar manner. Their data could be used for PFCR, but the authors worked on classification instead.

PFCR datasets are summarized in Table 1. Our dataset has the highest number of fact-checked claims. It also has the second-highest number of input claims and pairs after _CrowdChecked_, but that dataset is significantly noisier. Finally, our dataset has by far the most languages, while the second biggest dataset in this regard has 5 language with only 50 samples per language.

Methods.Methods used for PFCR are usually either BM25 (and other similar information retrieval algorithms) or various text embedding-based approaches Vo and Lee (2018); Shaar et al. (2022a, b, i.a.). Reranking is often used to combine several methods to side-step compute requirements or as a sort of ensembling Shaar et al. (2020), i.a.). PFCR task is also a target of the _CLEF's CheckThat!_ challenge, with many teams contributing with their solutions Nakov et al. (2022). Other methods use visual information from images Mansour et al. (2022); Vo and Lee (2020), abstractive summarization Bhatnagar et al. (2022), or key sentence identification Sheng et al. (2021) to improve the results.

## 3 Our Dataset

Our dataset _MultiClaim_ consists of fact-checks, social media posts and pairings between them.

Fact-checks.We have collected the majority of fact-checks listed in the Google Fact Check Explorer, as well as fact-checks from additional manually identified major sources (e.g., Snopes) that were missing. Overall, we have collected 205,751 fact-checks from 142 fact-checking organizations covering 39 languages. We publish the _claim, title, publication date, and URL_ of each fact-check. We do not publish the full body of the articles. The claim is usually (in 88.2% of the cases) a one sentence long summarization of the information being fact-checked.

\begin{table}
\begin{tabular}{l r r r r} \hline \hline  & Input claims & FC claims & Pairs & Languages \\ \hline Kazemi et al., 2021 & N/A & 150,000 & 258 & 5 \\ Jiang et al., 2021 & N/A & 90 & 1,573 & 1 \\ Shaar et al., 2020 & N/A & 27,032 & 1,768 & 1 \\ Shaar et al., 2021 & 2,259 & 44,164 & 2,440 & 2 \\ Hardalov et al., 2022 & 316,564 & 10,340 & 332,660 & 1 \\ \hline _MultiClaim_ (our) & 28,092 & 205,751 & 31,305 & 27/39 \\ \hline \hline \end{tabular}
\end{table}
Table 1: PFCR datasets. FC claims are _fact-checked_. _NA_ means that we were not able to identify the correct number of input claims. The number should be similar to the number of pairs in most cases.

 

Social media posts.We used two ways to find relevant social media posts from Facebook, Instagram and Twitter. In both cases, it was professional fact-checkers that assigned the fact-checks to the posts. **(1)** Some fact-checks use the _ClaimReview_ schema2, which has a field for reviewed items. All the links to the three social media platforms from this field are used to collect the posts and form the _pairs_. **(2)** We searched for URLs to Facebook and Instagram in the main body of the fact-checks. This is our pool of potentially relevant posts. Then we use the fact-checking warnings these two platforms provide. These warnings contain links to relevant fact-checking articles. We use these links to establish additional _pairs_.

Footnote 2: [https://schema.org/ClaimReview](https://schema.org/ClaimReview)

In total, we collected 28,092 posts from 27 languages. There are 31,305 _fact-check-to-post pairs_, each post in our dataset is paired with at least one fact-check. 26,774 of these pairs are monolingual and 4,212 are crosslingual (as predicted by the language identification, see below). Figure 1 shows the major (more than 100 samples) languages. All the crosslingual cases have the visualized language for posts and English for fact-checks. We can see that there is a clear distinction between these two groups, probably caused by different fact-checking cultures in different regions.

We publish the _text, OCR of the attached images (if any), publication date, social media platform, and fact-checker's rating_ of each post. The _rating_ is the reason why the post was flagged (see Section 4.2 for more details). We do not publish URLs in an effort to protect the users and their privacy as much as possible. For detailed information about the implementation of this dataset collection pipeline, see Appendix B. For a more detailed breakdown of dataset statistics (by languages and sources), see Appendix C. Examples from our dataset can be seen in Appendix G.

Dataset versions.We machine-translated all the published texts into English, resulting in two parallel versions of our dataset: the _original version_ and the _English version_. We also used automatic language identification on all the texts. Both translations and language identifications are published as well.

Noise ratio.We manually checked 100 randomly selected pairs from our dataset and evaluated their validity. Three authors rated these pairs and assessed whether the claim from the fact-check was made in the post. In case of disagreement, they discussed the annotation until an agreement was reached. Based on our assessment, 87 out of 100 pairs were correct. The remaining 13 pairs were not errors made by social media platforms or fact-checkers, but rather posts that required visual information (either from video or image) to fully match the assigned fact-check. The 95% Agresti-Coull confidence interval Agresti and Coull (1998) for correct samples in our dataset is 79-92%.

## 4 Unsupervised Evaluation

We formulate the task we are solving with our dataset as a ranking task, i.e., for each post, the methods rank all the fact-checks. Then, we evaluate the performance based on the rank of the desired fact-checks by using success-at-K (S@K) as the main evaluation metric. We define as the percentage of pairs when the desired fact-check ends up in the top K. Throughout the paper, we report this metric with the \(95\%\) Agresti-Coull confidence interval.

For unsupervised evaluation, we evaluated text embedding models and the BM25 algorithm to understand how they are able to handle pairs in different languages or even corsslingual pairs. Fact-checks are represented with their claims only. Posts are represented with their main texts concatenated with the OCR transcripts. We use either the original texts or their English translations, depending on the version of the dataset that is reported.

Text embedding models (TEMs).We use various neural TEMs Reimers and Gurevych (2019) that encode texts into a vector space. These are usually based on pre-trained transformer language models fine-tuned as Siamese networks to generate

Figure 1: Major languages from our dataset. Crosslingual languages all have English fact-checks.

 

well-performing text embeddings. We use these models to embed both social media posts and fact-checked claims into a common vector space. The retrieval is then reduced to calculating and sorting distances between vectors.

BM25With BM25 [14], we use the posts as queries and fact-checked claims as documents. The score is then calculated based on the lexical overlap between the query and all the documents.

### Main Results

We compare the performance of 15 English TEMs, 5 multilingual TEMs, and BM25. The English TEMs were only evaluated with the _English_ version. The multilingual TEMs and BM25 were evaluated with both the _original_ and the _English_ versions. BM25 with different versions is denoted as BM25-Original and BM25-English, respectively.

In this section, we use different strategies to evaluate monolingual and crosslingual pairs. For monolingual pairs, we only search within the pool of fact-checks written in the same language as the post (e.g., for a French post we only rank the French fact-checks). For crosslingual pairs, we search in all the fact-checks3. In both cases, we report the average performance for individual languages. We only report for languages with more than 100 pairs. For crosslingual pairs, we also consider a separate _Other_ category for all the leftover pairs.

Footnote 3: The index created for BM25 is multilingual as well.

We present the main results in Table 2 and we visualize them in Figure 2. We conclude that: **(1)** English TEMs are the best performing option for both monolingual and crosslingual claim retrieval. **(2)** Machine translation significantly improved the performance of both BM25 and TEMs. The difference between the best performing _English_ version method and the best performing _original_ version method is 35% for crosslingual and 14% for monolingual S@10. Currently, machine translation systems also have better language coverage than multilingual TEMs. **(3)** TEMs have a strong correlation between monolingual and crosslingual performance (Pearson's \(\rho=0.98\), \(P=4e{-}10\) for English TEMs). These two capabilities do not conflict. **(4)** There is almost no correlation (Pearson's \(\rho=0.03\), \(P=0.89\) for English TEMs) between model size and performance. The training procedure is much more important. GTR is an exceptionally well-performing family, with all three models being Pareto optimal w.r.t. model size and performance. Another notable model is MiniLM - a surprisingly powerful model for its size (33M).

LanguagesPerformance for individual languages is shown in Figure 3. We show the results for the best performing TEMs for both versions (**GTR-T5-Large** for the _English_ and **MPNetBase-Multilingual** for the _original_, which are denoted as GTR-T5 and MPNet from now on) and both BM25s. We cannot directly compare the performance numbers across different monolingual languages, since they use different pools of fact-checks with different sizes. This is also why smaller languages seem to have better scores.

BM25-Original, despite its seemingly weak overall performance, is actually competitive in some languages, e.g., Spanish, Portuguese, or Malay. It is better than multilingual TEMs for 7 out of 20 monolingual cases. Its overall monolingual performance is significantly decreased by Thai and Myanmar, due to their use of _scriptio continua_. On the other hand, unlike multilingual TEMs, BM25-Original is by design not capable of any crosslingual retrieval and the results are shown only for completeness.

False positive rate.We noticed that BM25-Original seems to perform better for languages with larger fact-check pools. We conducted an

\begin{table}
\begin{tabular}{l c r r r r} \hline \hline Method & Size [M] & Ver. & Mono & Cross & SLB \\ \hline \multicolumn{6}{c}{**BM25**} \\  & En & 0.78 & 0.39 & 0.18 \\  & & Oz & 0.62 & 0.06 & 0.69 \\ \hline \multicolumn{6}{l}{**English TEMs**} \\  & Ditklb/BERT & K2 & En & 0.76 & 0.43 & 0.18 \\  & 110 & En & 0.81 & 0.51 & 0.19 \\  & GTR-T5-Large & 336 & En & **0.83** & **0.56** & 0.20 \\  & GTR-T5-XL & 1242 & En & **0.83** & **0.56** & 0.20 \\  & MPNetBase & 109 & En & 0.78 & 0.47 & 0.18 \\  & MSARC-BERT-base & 109 & En & 0.78 & 0.46 & 0.18 \\  & MiniLM-XLI2 & 33 & En & 0.80 & 0.48 & 0.18 \\  & MRDG+APNet-Base & 109 & En & 0.80 & 0.50 & 0.18 \\  & SGPT+12SM & 125 & En & 0.63 & 0.25 & 0.14 \\  & SOPT+2TE & 2700 & En & 0.77 & 0.50 & 0.19 \\  & Sentence T-S Base & 110 & En & 0.73 & 0.57 & 0.14 \\  & Sentence T-ST Large & 336 & En & 0.75 & 0.41 & 0.15 \\  & Sentence T-SLAr & 1242 & En & 0.78 & 0.47 & 0.16 \\ \hline \multicolumn{6}{l}{**Multilingual TEMs**} \\  & DistilUSE-Basic-Multilingual & 135 & En & 0.74 & 0.40 & 0.15 \\  & & Og & 0.66 & 0.20 & 0.16 \\ LaBSE & 472 & En & 0.63 & 0.22 & 0.13 \\  & Qz & 0.69 & **0.21** & 0.17 \\ MPINet-Base-Multilingual & 278 & En & 0.75 & 0.41 & 0.16 \\  & & Op & **0.79** & 0.21 & 0.17 \\  & & 118 & Dra & 0.74 & 0.38 & 0.16 \\  & & Og & 0.63 & 0.15 & 0.17 \\ XLM-R & 278 & En & 0.72 & 0.33 & 0.15 \\  & & Og & 0.66 & 0.15 & 0.16 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Results for methods showing both _mon_inqual and _cross_lingual S@10. Ver. denotes either the _original_ (Og) or the _English_ (En) version of our dataset. The best results for these two versions are bolded. SLB denotes _same language bias_.

 

experiment to measure how pool size affects the results. We randomly selected 100 pairs for 7 of our languages with the largest fact-check pools. We then measured the performance for these 100 pairs while increasing the pool size from 100 to 2,100 by gradually adding random fact-checks.

We found that our initial observation was correct and that BM25-Original performs better than the MPNet model as the pool size increases (especially for Spanish, Portuguese, and French). The relative comparison between BM25 methods and TEMs is shown in Figure 4. This suggests, that MPNet has a higher _false positive rate_, i.e., it is more likely to assign high scores to irrelevant fact-checks. As the number of fact-checks grows, the risk of selecting irrelevant fact-checks also grows. **Different methods may be appropriate for different languages based on the number of fact-checks available.** We did not find the same pattern when comparing the methods using the _English_ version.

Same language bias.The fact that we reduce the fact-checks pool to one language in monolingual evaluation is motivated by what we call _same language bias_ (SLB) - a tendency of methods to retrieve fact-checks that have the same language as the post. We approximate SLB by calculating the percentage of top 10 fact-checks that have the same language as the input post when we use the full pool. This number is reported in Table 2.

BM25-Original has the highest SLB score of all the methods, as it has an implicit language filter that effectively removes fact-checks from other languages from the pool. This reduction makes the task easier, but it violates our requirement that the method should take fact-checks in all the languages into consideration. We used language-filtered fact-checks in monolingual evaluation to reduce the effect the SLB has on the results. Without this filtering, BM25-Original would clearly outperform MPNet (S@10 \(51.9\) vs \(38.5\)), even though our results in Figure 3 show that for many languages, its language understanding capabilities are actually worse.

However, it is not necessarily true that a higher SLB leads to worse crosslingual performance. As shown in Figure 5, TEMs with the highest SLB actually have the best performance for crosslingual evaluation. Even more strikingly, the relative crosslingual performance compared to monolingual performance increases with SLB as well. We theorize that a certain amount of SLB is healthy, as long as the methods focus on meaningful similarities in texts written in the same language, such as local topics, named entities, and events, rather than on superfluous lexical overlaps. SLB can also be useful to localize claims that are not specific enough. For example, it is impossible to identify the country of origin for the following claim translated to English: _Educational institutions are reopening from January 18_. However, as soon as we know that the original language was Bengali, we can guess that it is about Bangladesh institutions.

### Other Dimensions

In this section, we report results for various data splits. Since we often work with small splits, we are not able to report the results as an average per language as in the previous section. Instead, we report the average score across the samples. This will give more weight to the more common languages, penalizing the methods with high _false positive rate_ (e.g., multilingual TEMs).

Time.We grouped the posts for which we were able to obtain the publication date (N = 26,337) into 20-quantiles and measured the performance of individual methods. The results are shown in Figure 6. There is a visible drop-off for all the methods at the start of 2020, largely caused by the COVID-19 pandemic. We confirmed this by measuring how well the methods worked on posts

Figure 2: Comparison of different method families. Unless stated otherwise, the methods use _English_ version of our dataset.

 

with the substrings _corona_, _covid_ or _korona_.4 The results are shown in Table 3 (top panel).

Footnote 4: We chose this as a very simple, high-precision filtering technique. Many other COVID-19-related posts were not retrieved.

The relative differences between individual methods seem stable. We hypothesized that TEMs might have problems with aging, since many of the foundation language models were originally trained before 2020. We correlated the average post time for each quantile with the difference between GTR-T5 and BM25-English performance and found a negative, but statistically insignificant correlation (Pearson \(\rho=-0.33\), \(P=0.17\) for monolingual S@10). Similar results were measured for crosslingual performance. In both cases, the direction signals that the GTR model is indeed getting worse over time. We found no such signal comparing methods using the _original_ version.

There is a risk that the fact-check was written based on the very post we are using, and an information leak might have happened (e.g., the fact-checker might have used parts of the post verbatim). To test this, we compared pairs where the post is newer with the pairs where the post is older. We found that the two groups have virtually the same performance for all the methods (e.g., 80.02 vs 80.04 monolingual S@10 for GTR-T5). If there is an information leak happening, we were not able to measure it.

Post rating.In the case of Facebook and Instagram posts, fact-checkers use the so-called _ratings_ to describe the type of fallacy present. We show the results for the most common ratings in Table 3 (middle panel). _Missing context_ has a slightly lower score than _(Partially) False information_. This might be caused by the fact that the rating is defined by what is _not_ written in the post, making it harder

Figure 4: Relative performance (S@10) between BM25 methods and TEMs for different fact-check pool sizes. For both versions we compare the best performing TEMs (GTR-T5 and MPNet) with BM25. Positive \(\rho\) means that BM25 gets better with the growing pool size.

Figure 5: Relation between _same language bias_ and performance for TEMs.

Figure 3: Performance of selected methods for individual languages. For crosslingual pairs (e.g., _eng-hin_), the first language is for the fact-checks and the second is for the posts. The number of pairs is shown at the top.

Figure 6: Performance of selected methods for posts from different time intervals. Shaded areas are confidence intervals.

 

to match with an appropriate fact-check. _Altered photo / video_ rating has an even lower score. This is an expected behavior, since our purely text-based models cannot handle cases when the crux of the post is in its visual aspect.

Post length.We show how the length of the posts influence the results in Figure 7. In general, the performance peaks at around 500 characters. Posts that are too short are too difficult to match (and extremely short posts may even indicate noise in the data). On the other hand, for posts longer than 500 characters, the methods gradually lose their effectiveness. The relative performance of methods seems to be relatively stable.

Social media platforms.The results for social media platforms are in Table 3 (bottom pannel). We can see that Twitter has the best performance overall. We believe that this is, to a large extent, caused by the limited length of the Twitter posts.

## 5 Supervised Training

To validate that our dataset can be used as a training set, we fine-tuned TEMs and evaluated their performance. We split the posts randomly into 80:10:10% train, development, and test sets. We used _cosine_ or _contrastive_ training losses to fine-tune the models. In both cases, both positive and negative pairs are required for training. We used our data as positive samples and random pairs as negative samples. We performed a hyperparameter search with GTR-T5 and MPNet TEMs (see Section D). Here, we report the best performing fine-tuned model we were able to achieve for both TEMs.

The overall results for the test set are reported in Table 4. We can see that GTR-T5 achieved only modest improvements6. On the other hand, MPNet improved significantly in both monolingual and crosslingual performance, even surpassing the performance of BM25-English. We observed that the improvements were global across all languages.

Footnote 6: This is largely caused by the fact that we had to use smaller batch size due to (1) this model being larger, and (2) this model not supporting _average mixed precision_. We believe that with larger batch size the performance could be even better.

We also observed that the TEMs were able to saturate the training set quite quickly, achieving 99.5%+ average precision after only a few epochs. This shows that our naive random selection of negative samples was too easy. The model can learn only a limited amount of information from such samples, and we would need a more elaborate scheme for generating more challenging negative samples. This could lead to further performance improvements.

## 6 Post-Hoc Results Analysis

The pairs, we obtained from the fact-checks, are only a subset of all the potentially valid pairs. This incompleteness limits our understanding of the dataset and also our evaluation. We decided to manually annotate a subset of the results generated by the methods to better understand what is missing from our data. We generated the top 10 fact-checks for the 87 test set posts that we knew had valid fact-checks (see Section 3). We used the 4 unsupervised and 2 supervised methods from Section 5.

These methods generated 3,390 unique pair predictions for these 87 posts. Three authors went through each prediction and marked, whether they agreed with it, i.e., whether they found the fact-check to be valid and useful for the post. The agreement rates between the annotators were sufficiently high: 82.2%, 85.5% and 92.9%. We consider pairs where at least two annotators agreed to be _correct_. In total, the methods were able to find 719 correct pairs. 96 of these were present in our original dataset. This suggests that there is roughly \(7\times\) more pairs in our dataset than we had previously identified. No method was able to find 9 fact-checks out of 105 that were already in our dataset. Of the 719 correct pairs, only 247 were monolingual, 136 were crosslingual with an English fact-check, and 336 were crosslingual with a non-English fact-check. The last category in particular is almost completely missing from our dataset.

Figure 7: Performance of selected methods for posts with different lengths. Shaded areas are confidence intervals.

 

In Table 4, we show the results for individual methods. We compare S@10 (now defined as how many posts have at least one correct fact-check produced in the top 10) as approximated with our dataset and the true S@10 obtained by the annotation. We can see that the score for our dataset is significantly lower and the true performance of our methods is better then what was measured previously. We also compare recall-at-10 (R@10), defined as the percentage of expected pairs a method was able to produce in the top 10. In this case, both our dataset and manual annotation are only estimates, since they do not contain _all_ the valid pairs, they both contain only a subset obtained by different methods. Here we can see that our dataset actually provides higher estimates. We assume that our annotation is more precise, so we conclude that the recall calculated from our dataset is overinflated (possibly due to selection bias). **It also seems that our dataset has a bias in favor of BM25**, compared to the results obtained from annotated data.

## 7 Discussion

Complexity of crosslingual evaluation.Phenomena such as _same language bias_ or _false positive rate_ make the evaluation of multilingual and crosslinkingal datasets inherently complex. If we were to abstract the whole evaluation into a single number, as is often done in practice, we would have completely missed these pitfalls. Without an in-depth evaluation, we might have been misled while applying our methods in practice, e.g., while developing helpful tools for fact-checkers. Our evaluation procedures were previously impossible to develop in the absence of linguistically diverse PFCR datasets.

Machine translation beats multilingual TEMs.These two technologies represent the two main multilingual and crosslingual learning paradigms - label transfer and parameter transfer Pikuliak et al. (2021). Machine translation is a clear winner in our case. English TEMs significantly outperform multilingual approaches for both monolingual and crosslingual retrieval.

Covid-19.As shown in Table 3, it seems that the performance for COVID-19 is significantly worse than for the rest of the dataset. However, this might not necessarily mean that the methods are having issues with the domain shift. The sheer amount of fact-checks written about COVID-19 makes it hard for the methods to pick the desired fact-check in the presence of thousands of other very similar ones. This is evident considering that BM25 also has worse results, even though it should be less prone to domain shift based on its design.

\begin{table}
\begin{tabular}{l|c c|c c|c c} \hline \hline  & \multicolumn{3}{c|}{Section 5 (S@10)} & \multicolumn{2}{c|}{Section 6 (S@10)} & \multicolumn{2}{c}{Section 6 (R@10)} \\ Model & Monolingual & Crosslingual & Our dataset & Annotated & Our dataset & Annotated \\ \hline
**Unsupervised** & & & & & & \\ GTR-T5-Large & \(0.82\pm 0.01\) & \(0.55\pm 0.05\) & \(0.70\pm 0.09\) & \(0.93\pm 0.05\) & \(0.69\pm 0.09\) & \(0.59\pm 0.04\) \\ BM25-English & \(0.74\pm 0.02\) & \(0.40\pm 0.05\) & \(0.67\pm 0.10\) & \(0.85\pm 0.07\) & \(0.67\pm 0.09\) & \(0.48\pm 0.04\) \\ MPNet-Base-Multilingual & \(0.63\pm 0.02\) & \(0.23\pm 0.04\) & \(0.51\pm 0.10\) & \(0.70\pm 0.09\) & \(0.47\pm 0.09\) & \(0.32\pm 0.03\) \\ BM25 Original & \(0.68\pm 0.02\) & \(0.09\pm 0.03\) & \(0.60\pm 0.10\) & \(0.71\pm 0.09\) & \(0.58\pm 0.09\) & \(0.26\pm 0.03\) \\ \hline
**Supervised** & & & & & & \\ GTR-T5-Large & \(0.84\pm 0.01\) & \(0.59\pm 0.05\) & \(0.71\pm 0.09\) & \(0.92\pm 0.05\) & \(0.70\pm 0.09\) & \(0.65\pm 0.03\) \\ MPNet-Base-Multilingual & \(0.76\pm 0.02\) & \(0.42\pm 0.05\) & \(0.62\pm 0.10\) & \(0.85\pm 0.07\) & \(0.60\pm 0.09\) & \(0.45\pm 0.04\) \\ \hline \hline \end{tabular}
\end{table}
Table 4: Test set performance (Section 5) and annotated results performance (Section 6) of unsupervised and supervised methods.

\begin{table}
\begin{tabular}{l|c c c c c c c c c} \hline \hline  & \multicolumn{3}{c|}{Manological} & \multicolumn{3}{c|}{Conceptual} & \multirow{3}{*}{MNpt} & \multicolumn{3}{c}{BMS25-Eng} & \multirow{3}{*}{MPNet} & \multicolumn{3}{c}{BM25-pg} \\  & \(N\) & GTR-T5 & BMS25-pn & MPNet & BMS25-pg & \(N\) & GTR-T5 & BMS25-pg & MFNet & BMS25-pg \\ \hline COVID-related & 4.59 & \(0.72\pm 0.01\) & \(0.68\pm 0.01\) & \(0.50\pm 0.02\) & \(0.60\pm 0.01\) & \(514\) & \(0.40\pm 0.04\) & \(0.29\pm 0.04\) & \(0.17\pm 0.03\) & \(0.06\pm 0.02\) \\ Otherwise & 20515 & \(0.83\pm 0.00\) & \(0.75\pm 0.01\) & \(0.66\pm 0.01\) & \(0.70\pm 0.01\) & \(3698\) & \(0.35\pm 0.02\) & \(0.39\pm 0.02\) & \(0.23\pm 0.01\) & \(0.08\pm 0.01\) \\ \hline False information & 14812 & \(0.82\pm 0.01\) & \(0.75\pm 0.01\) & \(0.65\pm 0.01\) & \(0.69\pm 0.01\) & \(215\) & \(0.52\pm 0.02\) & \(0.37\pm 0.02\) & \(0.22\pm 0.02\) & \(0.09\pm 0.01\) \\ Party type like information & 4498 & \(0.82\pm 0.02\) & \(0.75\pm 0.01\) & \(0.63\pm 0.01\) & \(0.70\pm 0.01\) & \(569\) & \(0.53\pm 0.04\) & \(0.39\pm 0.04\) & \(0.21\pm 0.03\) & \(0.08\pm 0.02\) \\ Missing context & 1993 & \(0.77\pm 0.02\) & \(0.70\pm 0.02\) & \(0.61\pm 0.02\) & \(0.65\pm 0.02\) & \(268\) & \(0.53\pm 0.06\) & \(0.35\pm 0.06\) & \(0.19\pm 0.05\) & \(0.05\pm 0.03\) \\ Altered photo/video & 753 & \(0.73\pm 0.03\) & \(0.66\pm 0.03\) & \(0.52\pm 0.04\) & \(0.64\pm 0.03\) & \(142\) & \(0.47\pm 0.08\) & \(0.34\pm 0.08\) & \(0.17\pm 0.06\) & \(0.12\pm 0.05\) \\ \hline Facebook & 24668 & \(0.81\pm 0.00\) & \(0.74\pm 0.01\) & \(0.64\pm 0.01\) & \(0.68\pm 0.01\) & \(3927\) & \(0.52\pm 0.02\) & \(0.37\pm 0.02\) & \(0.22\pm 0.01\) & \(0.08\pm 0.01\) \\ Instagram & 1473 & \(0.78\pm 0.02\) & \(0.74\pm 0.02\) & \(0.36\pm 0.03\) & \(0.75\pm 0.02\) & \(44\) & \(0.56\pm 0.14\) & \(0.37\pm 0.14\) & \(0.19\pm 0.11\) & \(0.19\pm 0.11\) \\ Twitter & 652 & \(0.84\pm 0.03\) & \(0.74\pm 0.03\) & \(0.69\pm 0.03\) & \(0.70\pm 0.03\) & \(244\) & \(0.64\pm 0.06\) & \(0.49\pm 0.06\) & \(0.38\pm 0.06\) & \(0.06\pm 0.03\) \\ \hline Total & 26774 & \(0.81\pm 0.00\) & \(0.74\pm 0.01\) & \(0.64\pm 0.01\) & \(0.68\pm 0.01\) & \(4212\) & \(0.53\pm 0.02\) & \(0.38\pm 0.01\) & \(0.23\pm 0.01\) & \(0.08\pm 0.01\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Performance (S@10) with confidence intervals for various splits and methods.

 

Conclusions

In this paper, we introduced a new multilingual _previously fact-checked claim retrieval_ dataset. Our collection process yielded a unique and diverse dataset with a relatively small amount of noise in it. We believe that the evaluation of various methods is also insightful and can lead to the development of better fact-checking tools in the future.

We believe that our dataset opens up many interesting research directions. We have, for example, barely scraped the surface of crosslingual learning in this work. Applying various transfer learning methods (especially for low-resource languages) is an important future direction.

## 9 Limitations

### Dataset

Noise.Based on our annotation (see Section 3), we expect that \(\sim\) 13% of posts in our dataset do not contain the claim in the textual form. These are the cases when the claim being made on the social media is based on visual information. Note, that the methods might still be able to retrieve correct fact-checks for some of these posts, based on spurious correlations, e.g., overlaps in named entities.

AI APIs.We use out-of-the-box AI services to perform optical character recognition, machine translation to English and language detection. All of these have limited precision and might inject noise into our data.

* _OCR_ was too sensitive and was often reading imaginary character, watermarks, etc. We had to address this by a more aggressive text cleaning.
* _Machine translation to English_ is not perfect and the quality of translations depends on source language, particular topics or even the writing style.
* _Language detection_ is an important component in our pipeline as we use it to group samples by language and then reason about these languages. Noise in language detection might have influenced our results and insights.

Selection bias.There is a possibility that selection bias influences our results. First, sometimes fact-checkers writing the fact-checks base their writing on a particular post and the fact-check might contain parts of it verbatim. We tried to measure the size of this effect by comparing cases when the fact-checks are newer and older then posts (see Section 4.2), but we did not find a signal that this is the case. However, we know that there are at least few samples with this problem.

Second, there might be a bias towards social media posts that the social media platform or fact-checkers are already able to detect. Other, more difficult cases might still elude us.

Linguistic bias.Although our dataset is quite diverse, compared to most published datasets, there is still a bias towards major languages and Indo-European language family in particular. Crosslingual pairs consist mostly of East or South Asian posts with non-Latin script mapped to English fact-checks. It is hard to estimate how our results would generalize to other language pairs. We visualize the languages in Figure 1. The annotation efforts in Section 6 show that there are many crosslinking pairs that our data collection methodology was not able to collect.

### Methods

Language support.The methods we use have different degrees of support for different languages. BM25 requires a proper tokenization to work. We have languages that use _scriptio continua_ - Thai and Myanmar - where this is a problem. BM25-Original performance for these two is subpar, but could be improved by implementing custom tokenization models.

Multilingual TEMs we use do not support Sinhala and Tagalog languages, i.e., they were not trained with their data. The performance for these two languages is again subbar. Additionally, all methods depending on machine translation are naturally only able to handle languages that have a machine translation system available, although we believe that this was not a significant problem in our dataset.

Hidden positive pairs.The results we report might be deflated from the practical point of view because of unmarked correct pairs that are in the dataset. We have information only about a small subset of all the pairs. Our attempt to approximate true performance is provided in Section 6.

Supervised learning overfitting.It is possible that our supervised training yielded model that is overfitted on the particular languages and time 

frame that are represented in our dataset. The increase in performance might not transfer to out-of-domain pairs.

## 10 Ethical Considerations

We analyzed the likelihood and impact of ethical and societal risks for the most affected stakeholders, such as social media users and profile owners, fact-checkers, researchers, or social media platforms. For the most severe risks, we proposed respective countermeasures, following the guidelines and arguments in Franzke et al. (2020); Townsend and Wallace (2016); Mancosu and Vegetti (2020).

Data collection processWhile Twitter posts were collected using a (at the time of collection) publicly available API, the Terms of Service (ToS) of Facebook and Instagram do not currently allow for the accessing or collecting of data using automated means. Following the discussion and arguments presented in Mancosu and Vegetti (2020) and to minimize the harm to these social media platforms and their users, we made sure to only collect publicly available posts that are accessible even without logging in. Even if we admit the risk that such research activities could potentially violate the ToS, we argue that ignoring posts from Facebook and Instagram would prohibit research that seeks to address key current issues such as disinformation on these platforms Bruns (2019). These are some of the main platforms for disinformation dissemination in many countries. We consider the collection of such public data and its usage for research purposes to be in the public interest, especially considering the status of disinformation as a hybrid security threat ENISA (2022), which could justify minor harms to social media platforms.

Other considerable risks include the risk of accessibility privacy intrusion Tavani (2016) of social media users by observing them in an environment where they do not want to be observed. We did not obtain explicit consent from social media users to collect their posts. However, the criteria for considering social media data private or public depend on the assumption of whether social media users can reasonably expect to be observed by strangers Townsend and Wallace (2016). Twitter is considered an open platform. The collected posts on Facebook or Instagram are not only public, but the users can also expect that their posts will be widely shared, commented or reacted to and they can end up being fact-checked if it is the case.

Data publicationTo minimize the risk of third-party misuse, the dataset is available only to researchers for research purposes. The full texts of the fact-checks are not published to avoid possible copyright violations.

We assessed the risk of re-identification, as well as the risk of revealing incorrect, highly sensitive or offensive content regarding social media users. At the same time, we had to take into account the fact that social media platforms remove some posts after they have been flagged as disinformation. Therefore, we decided to include the original texts of the posts in the dataset to prevent it from decaying. Otherwise, it would become progressively less usable and research based on it less reproducible. This also allows us to avoid publishing the URLs of posts, which would directly reveal the identities of the users. It is not possible to guarantee complete anonymity, since the posts are still linked in the fact-checks. The posts could also theoretically be found by full-text search.

On the other hand, all the posts released in our dataset are already mentioned in a publicly available space in the context of fact-checking efforts. Our publication of these posts does not significantly increase their already existing public exposure, especially considering the limited access options of our dataset.

To support users' rights to rectification and erasure in case of the publication of incorrect or sensitive information, we provide a procedure for them to request the removal of their posts from the dataset. However, we assess that the risk of wrongfully assigned fact-checks has a low probability (see Section 3).

As the dataset can also be used for supervised training (see Section 5), there is a risk of propagating biases present in the data (see Section 9). We recommend performing a proper linguistic analysis of any supervised model w.r.t. all the languages for which the model is intended. The results shown in this paper may not reflect the performance of the methods on other languages. We are also aware of the risk of propagating the biases of the fact-checkers, as it is they who decide what to fact-check. Although they should generally follow principles of fact-checking ethics (see, e.g., the IFCN's Code of Principles), there may still be present some human or systemic biases Schwartz et al. (2022) that could affect the results when using the dataset for other purposes 



## Acknowledgements

This work was partially supported by the _Central European Digital Media Observatory (CEDMO)_, a project funded by the European Union under the Contract No. 2020-EU-IA-0267, and by _DisAI - Improving scientific excellence and creativity in combating disinformation with artificial intelligence and language technologies_, a project funded by European Union under the Horizon Europe, GA No. 101079164.

We thank _Weights & Biases_ for providing us a license for their experiment tracking platform.

## References

* Agresti and Coull [1998] Alan Agresti and Brent A Coull. 1998. Approximate is better than "exact" for interval estimation of binomial proportions. _The American Statistician_, 52(2):119-126.
* Barron-Cedeno et al. [2020] Alberto Barron-Cedeno, Tamer Elsayed, Preslav Nakov, Giovanni Da San Martino, Maram Hasanain, Reem Suwaileh, Fatima Haouari, Nikolay Babulkov, Bayan Hamdan, Alex Nikolov, Shaden Shaar, and Zien Sheikh Ali. 2020. Overview of CheckThat! 2020: Automatic Identification and Verification of Claims in Social Media. In _Experimental IR Meets Multilinguality, Multimodality, and Interaction_, volume 12260 of _Lecture Notes in Computer Science_, pages 215-236, Cham. Springer International Publishing.
* Bhatnagar et al. [2022] Varad Bhatnagar, Diptesk Kanojia, and Kimeswari Chebrolu. 2022. Harnessing abstractive summarization for fact-checked claim detection. In _Proceedings of the 29th International Conference on Computational Linguistics_, pages 2934-2945, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
* Blei et al. [2003] David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent Dirichlet allocation. _Journal of Machine Learning Research_, 3(Jan):993-1022.
* Bruns [2019] Axel Bruns. 2019. After the 'APIcalypse': social media platforms and their fight against critical scholarly research. _Information, Communication & Society_, 22(11):1544-1566.
* NORdic observatory for digital media and information DISorders.
* ENISA [2022]ENISA. 2022. ENISA Threat Landscape 2022.
* Franzke et al. [2020] Aline Shakki Franzke, Anja Bechmann, Michael Zimmer, Charles Ess, and the Association of Internet Researchers. 2020. Internet research: Ethical guidelines 3.0.
* Fact [2020] Full Fact. 2020. The challenges of online fact checking.
* Grootendorst [2022] Maarten Grootendorst. 2022. BERTopic: Neural topic modeling with a class-based TF-IDF procedure. arXiv:2203.05794 [cs.CL].
* Hardalov et al. [2022] Momchil Hardalov, Anton Chernyavskiy, Ivan Koychev, Dmitry Ilvovsky, and Preslav Nakov. 2022. CrowdChecked: Detecting previously fact-checked claims in social media. In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 266-285, Online only. Association for Computational Linguistics.
* Hrckova et al. [2022] Andrea Hrckova, Robert Moro, Ivan Srba, Jakub Simko, and Maria Bielikova. 2022. Automated, not automatic: Needs and practices in European fact-checking organizations as a basis for designing human-centered AI systems. arXiv:2211.12143 [cs.CY].
* Jiang et al. [2021] Ye Jiang, Xingyi Song, Carolina Scarton, Ahmet Aker, and Kalina Bontheva. 2021. Categorising Fine-to-Coarse Grained Misinformation: An Empirical Study of COVID-19 Infodemic. arXiv:2106.11702 [cs].
* Kazemi et al. [2021] Ashkan Kazemi, Kiran Garimella, Devin Gaffney, and Scott Hale. 2021. Claim matching beyond English to scale global fact-checking. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 4504-4517, Online. Association for Computational Linguistics.
* Mancosu and Vegetti [2020] Moreno Mancosu and Federico Vegetti. 2020. What You Can Scripe and What Is Right to Scrape: A Proposal for a Tool to Collect Public Facebook Data. _Social Media + Society_, 6(3). SAGE Publications Ltd.
* Mansour et al. [2022] Watheq Mansour, Tamer Elsayed, and Abdulaziz Al-Ali. 2022. Did I See It Before? Detecting Previously-Checked Claims over Twitter. In _Advances in Information Retrieval_, Lecture Notes in Computer Science, pages 367-381, Cham. Springer International Publishing.
* Maclaflef et al. [2022] Nicholas Micallef, Vivienne Armacost, Nasir Memon, and Sameer Patil. 2022. True or false: Studying the work practices of professional fact-checkers. _Proc. ACM Hum.-Comput. Interact._, 6(CSCW1).
* Nakov et al. [2022] Preslav Nakov, Alberto Barron-Cedeno, Giovanni da San Martino, Firoj Alam, Julia Maria Strub, Thomas Mandl, Ruben Miguez, Tommaso Caselli, Mucahid Kuttu, Wajdi Zaghouani, Chengkai Li, Shaden Shaar, Gautam Kah Shore Shah, Handy Mubarak, Alex Nikolov, Nikolay Babulkov, Yavuz Selim Kartal, Michael Wiegand, Melanie Siegel, and Juliane Kohler. 2022. Overview of the CLEF-2022 CheckThat! Lab on Fig

the COVID-19 infodemic and fake news detection. In _Experimental IR Meets Multilinguality, Multimodality, and Interaction_, pages 495-520, Cham. Springer International Publishing.
* Nakov et al. (2021) Preslav Nakov, David Corney, Maram Hasanain, Firoj Alam, Tamer Elsayed, Alberto Barron-Cedeno, Paolo Papotti, Shaden Shaar, and Giovanni Da San Martino. 2021. Automated Fact-Checking for Assisting Human Fact-Checkers. In _Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI-21)_, pages 4551-4558. International Joint Conferences on Artificial Intelligence Organization.
* Pikuliak et al. (2021) Matus Pikuliak, Marian Simko, and Maria Bielikova. 2021. Cross-lingual learning for text processing: A survey. _Expert Systems with Applications_, 165.
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence BERT: Sentence embeddings using Siamese BERT-networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
* Robertson and Zaragoza (2009) Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. _Found. Trends Inf. Retr._, 3(4):333-389.
* Schwartz et al. (2022) Reva Schwartz, Apostol Vassilev, Kristen Greene, Lori Perine, Andrew Burt, and Patrick Hall. 2022. Towards a standard for identifying and managing bias in artificial intelligence. NIST Special Publication 1270, National Institute of Standards and Technology.
* Shaar et al. (2022a) Shaden Shaar, Firoj Alam, Giovanni Da San Martino, and Preslav Nakov. 2022a. The role of context in detecting previously fact-checked claims. In _Findings of the Association for Computational Linguistics: NAACL 2022_, pages 1619-1631, Seattle, United States. Association for Computational Linguistics.
* Shaar et al. (2020) Shaden Shaar, Nikolay Babulkov, Giovanni Da San Martino, and Preslav Nakov. 2020. That is a known lie: Detecting previously fact-checked claims. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 3607-3618, Online. Association for Computational Linguistics.
* Shaar et al. (2022b) Shaden Shaar, Nikola Georgiev, Firoj Alam, Giovanni Da San Martino, Aisha Mohamed, and Preslav Nakov. 2022b. Associating the human fact-checkers: Detecting all previously fact-checked claims in a document. In _Findings of the Association for Computational Linguistics: EMNLP 2022_, pages 2069-2080, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Conference and Labs of the Evaluation Forum_, volume 2936. CEUR-WS.
* Sheng et al. (2021) Qiang Sheng, Juan Cao, Xueyao Zhang, Xirong Li, and Lei Zhong. 2021. Article reranking by memory-enhanced key sentence matching for detecting previously fact-checked claims. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Association for Computational Linguistics.
* ROME 2019, colocated with SIGIR 2019_.
* Tavani (2016) H.T. Tavani. 2016. _Ethics and Technology: Controversies, Questions, and Strategies for Ethical Computing_, 5th edition. Wiley.
* Townsend and Wallace (2016) Leanne Townsend and Claire Wallace. 2016. Social media research: A guide to ethics.
* Vo and Lee (2018) Nguyen Vo and Kyumin Lee. 2018. The Rise of Guardians: Fact-checking URL Recommendation to Combat Fake News. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_, SIGIR '18, pages 275-284, New York, NY, USA. Association for Computing Machinery.
* Vo and Lee (2020) Nguyen Vo and Kyumin Lee. 2020. Where are the facts? Searching for fact-checked information to alleviate the spread of fake news. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7717-7731, Online. Association for Computational Linguistics.

## Appendix A Computational Resources

We calculated all the results on an AWS-based virtual machine located in the Ohio AWS data center. The machine has one NVIDIA Tesla T4 GPU installed. The unsupervised experiments would take approximately 2 GPU days to replicate. The supervised experiments would take approximately 3 GPU days to replicate. Additional roughly 4 GPU days were spent on other experiments that were discarded or are not reported in this paper 

Dataset Pipeline Details

### Dataset Collection

The dataset was collected via our research platform _Monant_(Sroba et al., 2019).

Crawling.We use a _Selenium-_based web crawler that visits the links, extracts the HTML content and parses it with the _Beautiful Soup6_ library.

Footnote 6: [https://www.crummy.com/software/BeautifulSoup/](https://www.crummy.com/software/BeautifulSoup/)

Noise in social media posts.We removed texts or OCR transcripts that we deemed noisy (shorter than 25 characters or more than 50% non-alphabetical characters). We then only kept posts where at least one text was considered not noisy. We also removed noisy lines from OCR transcripts (Lines shorter than 5 characters or with more than 50% non-alphabetical characters). We also removed URLs.

Post deduplication.We unified posts that ended up with identical text contents after the cleaning process.

Machine translation.We translated all the texts into English. The only exceptions were fact-check claims coming from English-language providers (e.g., Snopes) that we considered English by default, and fact-check claims where CLD37 identified English language. We confirmed experimentally that CLD3 has a high precision on English texts.

Footnote 7: [https://github.com/google/cld3](https://github.com/google/cld3)

Language identification normalization.We observed that there are some systematic errors in the language identification models we used. We found out that the model often selected less common languages based on spurious patterns, e.g., mentions of Filipino politicians sometimes led to Ilocano language prediction. Based on data analysis, we changed some predictions automatically, e.g., all Ilocano predictions were changed into English. Sometimes we only did it when the script did not match the language, e.g., for posts with Latin script identified as Oromo. We do not recommend using this process automatically on any data. In other contexts, the generated predictions might be less noisy. Even in our case, we have different rules for posts and fact-checks based on the characteristics of these two domains. If the predictions proved to be too noisy, we unified several languages or language varieties into one. This is the case of Croatian, Bosnian and Serbian, as well as Indonesian and Malay.

## Appendix C Dataset Statistics

We show the number of fact-checks and posts per language in Table 5. For fact-checks, we only take into consideration the language of claim, since we mostly only work with claims in this work.

 

Posts can have more than one language detected based on its overall compositions. We calculated percentage for each language based on the language prediction methods. We consider all languages with at least 20% to be relevant. 25,482 posts have only one language detected, while 2,549 has two, 59 has three, 1 has four and 1 has zero.

Table 6 shows the sources of our fact-checks. Here we only show the statistics for the fact-checks we actually used in our experiments. There are additional 6k fact-checks that we have not used because they we were not able to fill their _claim_ field.

Table 7 show the number of fact-check-to-post pairs for different language combinations.

Figure 8 show the density of lengths for both the fact-checked claims and the posts. Both have long tail distributions, but the claims are in general much shorter. 99% of claims are shorter than 379 characters. For social media posts, it is 4129 characters.

## Appendix D Hyperparameters

### Bm25

We use the default PyTerrier values for BM25 algorithm: \(k_{1}=1.2,b=0.75\). Our preliminary results show that the results are not very sensitive towards these two hyperparameters, probably because of the relatively short length of the documents that we retrieve. Most claims in our dataset have only one sentence.

### Supervised Training

Table 8 show the range of hyperparmeters in our hyperparameter search done for supervised training in Section 5, as well as the best performing hyperparameters.

[TABLE:A1.T5][ENDTABL

Figure 8: Density plots for the character lengths of the fact-checked claims and the social media posts in our dataset.

 



\begin{table}
\begin{tabular}{l l r|c|r c|c|c|c



## Appendix E Additional Results

### Additional Metrics

Table 9 show additional IR metrics calculated for the experiments done in Section 4.1. There is a strong correlation between all these metrics, as shown in Table 10. This is caused by the fact that most posts have only one fact-check assigned and the calculations for such cases are very similar for different metrics. We ultimately decided to use S@10 as our main evaluation metric in this work as we find it to be the most interpretable measure (_for how many pairs the expected fact-checked claim ended up in the top 10_).

### Detailed Per-language Results

Table 11 shows additional language-specific results for all the methods from Section 4 including confidence intervals.

## Appendix F Other Ideas

Here we discuss some additional ideas that were tried and that we decided not to include in the main text for various reasons.

Sliding window embedding.Figure 7 shows that the performance for methods decreases for posts with certain length. The decrease is generally starting at around 500 characters. We experimented with using sliding windows with various sizes (both based on the number of characters and the number of sentences) and strides. TEMs then encode only this sliding window and the final vector similarity is calculated as the maximum similarity of any of the windows. We found out that this technique can slightly (\(+0.01-0.02\) S@10) improve the results for TEMs.

Using fact-check titles alongside claims.We represent fact-checks with the _claim_ field obtained from the data in our main text experiments. We also experimented with the _title_ field that we were able to obtain for the majority of the fact-checks. We found out that representing the fact-check as a concatenation of a claim and a title improves the results slightly (\(+0.00-0.01\) S@10) for BM25 methods.

Topic detection.We attempted to run a topic detection over our posts to better understand how different methods handle different topics and themes in our data. We experimented with both _original_ and _English_ versions, with both multilingual and monolingual topic detection models, such as LDA Blei et al. (2003) or BERTopic Grootendorst (2022). Ultimately we were not content with the quality of topic detection, as the models failed to reliably identify even the most frequent topics in our data, such as the COVID-19 pandemic or Russo-Ukrainian war. We believe that this is caused by the short length of the majority of the posts, as well as their relatively noisy nature.

Mixing original and English versions.We experimented with representing both fact-checks and posts as a concatenation of both the original language texts and the English translations, so that the multilingual methods can use both sources of information. However, this increased the _same language bias_ significantly while the performance decreased significantly across the board.

## Appendix G Examples

This Appendix contains 5 randomly selected fact-check-post pairs from our dataset. We show here all the information present in our dataset for these samples.

\begin{table}
\begin{tabular}{l|c|r|r} \hline
**Hyperparameter** & **Range** & **GTR-TS-Large** & **MPNet-Base-Multilingual** \\ \hline Loss & contrastive cosine & online-contrastive & online-contrastive \\ Learning rate & \([1e{-3},1e{-7}]\) & \(1e{-5}\) & \(5e{-6}\) \\ Learning rate schedule & cosine linear & cosine & cosine \\ Warmup steps & \([100,3200]\) & 800 & 1600 \\ Weight decay rate & \([1e{-7},1e{-4}]\) & \(1e{-5}\) & \(8e{-5}\) \\ Ratio of positive to negative samples & \([10,50\%]\) & 10\% & 30\% \\ Margin & \([0.1,0.5]\) & 0.5 & 0.4 \\ Batch size & Maximum possible & 2 & 8 \\ \hline \end{tabular}
\end{table}
Table 8: Range of hyperparameters used in our supervised hyperparameter search and the hyperparameters of our most successful models. The ranges adjusted during the experimentation according to the preliminary results.





\begin{table}
\begin{tabular}{l c r r r r r r r r r r}  & \multicolumn{4}{c}{S@10} & \multicolumn{2}{c}{MRR} & \multicolumn{2}{c}{MAP} & \multicolumn{2}{c}{NDCG} & \multicolumn{2}{c}{MAP@10} \\ Method & Ver. & Mono & Cross & Mono & Cross & Mono & Cross & Mono & Cross & Mono & Cross \\ \hline
**BM25** & & & & & & & & & & & \\ BM25 & En & 0.61 & 0.22 & 0.78 & 0.39 & 0.61 & 0.22 & 0.68 & 0.33 & 0.82 & 0.40 \\ BM25 & Og & 0.48 & 0.04 & 0.62 & 0.06 & 0.47 & 0.04 & 0.56 & 0.11 & 0.65 & 0.07 \\ \hline
**English TEMs** & & & & & & & & & & \\ DistilRoBERTa & En & 0.59 & 0.24 & 0.76 & 0.43 & 0.59 & 0.24 & 0.67 & 0.36 & 0.80 & 0.44 \\ GTR-T5-Base & En & 0.65 & 0.30 & 0.81 & 0.51 & 0.64 & 0.30 & 0.71 & 0.41 & 0.85 & 0.53 \\ GTR-T5-Large & En & **0.67** & 0.33 & **0.83** & **0.56** & **0.67** & **0.33** & 0.73 & **0.45** & **0.88** & **0.58** \\ GTR-T5-XL & En & **0.67** & **0.34** & **0.83** & **0.56** & **0.67** & **0.33** & **0.74** & **0.45** & 0.87 & **0.58** \\ MPNet-Base & En & 0.61 & 0.27 & 0.78 & 0.47 & 0.60 & 0.27 & 0.68 & 0.38 & 0.82 & 0.48 \\ MSMARCO-BERT-Base & En & 0.62 & 0.26 & 0.78 & 0.46 & 0.61 & 0.26 & 0.69 & 0.37 & 0.82 & 0.48 \\ MiniLM-L12 & En & 0.64 & 0.29 & 0.80 & 0.48 & 0.63 & 0.29 & 0.71 & 0.40 & 0.84 & 0.50 \\ MultiQA-MPNet-Base & En & 0.64 & 0.29 & 0.80 & 0.50 & 0.63 & 0.29 & 0.71 & 0.41 & 0.84 & 0.52 \\ SGPT-125M & En & 0.47 & 0.14 & 0.63 & 0.25 & 0.47 & 0.14 & 0.56 & 0.25 & 0.66 & 0.26 \\ SGPT-2.7B & En & 0.60 & 0.29 & 0.77 & 0.50 & 0.60 & 0.29 & 0.68 & 0.40 & 0.81 & 0.52 \\ Sentence-T5-Base & En & 0.57 & 0.21 & 0.73 & 0.37 & 0.56 & 0.21 & 0.65 & 0.33 & 0.77 & 0.38 \\ Sentence-T5-Large & En & 0.58 & 0.24 & 0.75 & 0.41 & 0.58 & 0.23 & 0.66 & 0.35 & 0.78 & 0.43 \\ Sentence-T5-XL & En & 0.61 & 0.27 & 0.78 & 0.47 & 0.60 & 0.26 & 0.68 & 0.38 & 0.82 & 0.48 \\ \hline
**Multilingual TEMs** & & & & & & & & & & \\ DistilUSE-Base-Multilingual & En & 0.58 & 0.22 & 0.74 & 0.40 & 0.58 & 0.22 & 0.66 & 0.34 & 0.78 & 0.41 \\  & Og & 0.50 & 0.10 & 0.66 & 0.20 & 0.49 & 0.10 & 0.59 & 0.21 & 0.69 & 0.21 \\ LaBSE & En & 0.47 & 0.13 & 0.63 & 0.22 & 0.46 & 0.13 & 0.56 & 0.23 & 0.66 & 0.23 \\  & Og & 0.53 & 0.12 & 0.69 & 0.22 & 0.53 & 0.12 & 0.61 & 0.22 & 0.72 & 0.23 \\ MPNet-Base-Multilingual & En & **0.60** & **0.24** & **0.75** & **0.41** & **0.59** & **0.23** & **0.67** & **0.35** & **0.79** & **0.42** \\  & Og & 0.53 & 0.11 & 0.70 & 0.21 & 0.53 & 0.11 & 0.61 & 0.22 & 0.73 & 0.22 \\ MiniLM-L2-Multilingual & En & 0.58 & 0.22 & 0.74 & 0.38 & 0.57 & 0.22 & 0.66 & 0.34 & 0.77 & 0.40 \\  & Og & 0.48 & 0.08 & 0.63 & 0.15 & 0.47 & 0.08 & 0.57 & 0.18 & 0.67 & 0.16 \\ XLM-R & En & 0.55 & 0.19 & 0.72 & 0.33 & 0.55 & 0.19 & 0.63 & 0.30 & 0.76 & 0.34 \\  & Og & 0.50 & 0.08 & 0.66 & 0.15 & 0.50 & 0.08 & 0.59 & 0.18 & 0.70 & 0.16 \\ \hline \end{tabular}
\end{table}
Table 9: The results for different ranking methods. This table shows the same experiment as Table 2, but also calculates additional information retrieval metrics: MRR, MAP, NDCG, MAP@10.





### Example #1

#### g.1.1 Fact-check

**ID:** 104315

**Published at:** 2021-09-27 factcheck.afp.com

**Claim Original text:**_Photo shows meeting of five international intelligence agencies in Delhi_

**Translated text:**_Photo shows meeting of five international intelligence agencies in Delhi_

**Detected languages:** eng: 100.0%

**Title Original text:**_This photo shows a delegation-level meeting between Indian and Russian national security advisors_

**Translated text:**_This photo shows a delegation-level meeting between Indian and Russian national security advisors_

**Detected languages:** eng: 100.0%

#### g.1.2 Social Media Post

**ID:** 16806

**Published at:** Facebook 2021-09-16

**Verdicts:** Partly false information

**Main text Original text:**_Post Giri IyerNow in Delhi !!India RAWIsrael MOSSADAmerica CIARussia KGBEngland MI6First time ever that the top five intelligence agency of the world are sitting together for a high level meeting in Delhi. This is the power of new India_

**Translated text:**_Post Giri IyerNow in Delhi !!India RAWIsrael MOSSADAmerica CIARussia KGBEngland MI6First time ever that the top five intelligence agency of the world are sitting together for a high level meeting in Delhi. This is the power of new India_

**Detected languages:** eng: 100.0%

### Example #2

#### g.2.1 Fact-check ID:** 34296

**Published at:** 2019-10-29 factuel.afp.com

**Claim Original text:**_COMMENT TRAITER L'HEPATITE B PAR LES PLANTES_

**Translated text:**_HOW TO TREAT HEPATIIIS B WITH HERBS_

**Detected languages:** fra: 100.0%

**Title Original text:**_Non, cette boisson a base de papaye, de citron, de racines de cocotier et de moringa bouillis, ne guert pas l'hepatite B_

**Translated text:**_No, this drink made from boiled papaya, lemon, coconut palm roots and moringa does not cure hepatitis B_

**Detected languages:** fra: 100.0%

#### g.2.2 Social Media Post ID:** 11569

**Published at:** Facebook 2019-07-06

**Verdicts:** False information

**Main text Original text:**_HEPATITE BET Cet tres rassurant faite cette experience et revenue temoigne !!! L hepatic n'es qu'un vieux souvenirs appres !!REMEDES POUR TRIET ET ERADIQUER L'HEPATITE B DU CORPSL'hepattie B est une infection virale qui s'attaque au foie.Le virus se transmet par le Sang ou lors des rapports sexuels. En effet, les seules secretions ou liquides coroprels qui permettent de transmettre le virus sont le sang, le sperme,les secretions vaginales, la salive et les liquides issus d'une plaieIngredients Une Papaye non mur Les RACINES de Papayer Fennelle Les feuilles fraches de Papayer femelle racines de Moringa fuilles frachee Moringa 4 circrons a couper en deux. racines de cocotierPreparation-Mettez les tous dans la marmite, les feuilles en dernier position. Ajoutez e l'eau et faites bouillir le melange.Mode d'emploiBoire 2 a 3 verres par jour.Ajoutze de l'eau a chaquete fois et faites bouillir une fois par joint . Suivrez le traitement pendant un mois.Faites yous examine par um medecin et revenez temoigner .Bonne gaverison...Aimes ton prochain par le partage de ce messageLa Boutique

\begin{table}
\begin{tabular}{l r r r r r} \hline \hline  & S@10 & MRK & MAP & NDCG & MAP@10 \\ \hline S@10 & 1.000 & 0.956 & 0.956 & 0.993 & 1.000 \\ MRR & 0.966 & 1.000 & 1.000 & 0.998 & 0.988 \\ MLP & 0.956 & 1.000 & 1.000 & 0.998 & 0.988 \\ NDC & 0.993 & 0.998 & 0.998 & 1.000 & 0.995 \\ MAP@10 & 1.000 & 0.988 & 0.988 & 0.995 & 1.000 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Pearson correlation coefficient between different metrics as calculated in Table 9. Both monolingual and crosslingual scores are taken into consideration.





\begin{table}
\begin{tabular}{l l c r r r r r r r r r r} \hline  & Ver. & spa & eng & pers & pers & pers & pers & pers & pers & pers \\ \hline BM25 & En & 0.79 \(\pm\) 0.01 & 0.62 \(\pm\) 0.01 & 0.78 \(\pm\) 0.01 & 0.77 \(\pm\) 0.02 & 0.74 \(\pm\) 0.02 & 0.70 \(\pm\) 0.03 & 0.82 \(\pm\) 0.03 & 0.89 \(\pm\) 0.03 & 0.66 \(\pm\) 0.04 & 0.81 \(\pm\) 0.04 \\ BM25 & Og & 0.76 \(\pm\) 0.01 & 0.60 \(\pm\) 0.01 & 0.79 \(\pm\) 0.01 & 0.71 \(\pm\) 0.02 & 0.73 \(\pm\) 0.02 & 0.56 \(\pm\) 0.03 & 0.75 \(\pm\) 0.03 & 0.26 \(\pm\) 0.04 & 0.74 \(\pm\) 0.04 & 0.58 \(\pm\) 0.04 \\ DistBiRoBERTa & Em & 0.72 \(\pm\) 0.01 & 0.64 \(\pm\) 0.01 & 0.64 \(\pm\) 0.01 & 0.64 \(\pm\) 0.02 & 0.79 \(\pm\) 0.02 & 0.75 \(\pm\) 0.02 & 0.58 \(\pm\) 0.03 & 0.79 \(\pm\) 0.03 & 0.89 \(\pm\) 0.03 & 0.65 \(\pm\) 0.04 & 0.82 \(\pm\) 0.04 \\ GTR-TBase & En & 0.80 \(\pm\) 0.01 & 0.72 \(\pm\) 0.01 & 0.75 \(\pm\) 0.01 & 0.84 \(\pm\) 0.02 & 0.80 \(\pm\) 0.02 & 0.66 \(\pm\) 0.03 & 0.85 \(\pm\) 0.02 & 0.90 \(\pm\) 0.02 & 0.71 \(\pm\) 0.04 & 0.85 \(\pm\) 0.03 \\ GTR-TS-Large & En & 0.84 \(\pm\) 0.01 & 0.77 \(\pm\) 0.01 & 0.80 \(\pm\) 0.01 & 0.36 \(\pm\) 0.01 & 0.82 \(\pm\) 0.02 & 0.69 \(\pm\) 0.03 & 0.86 \(\pm\) 0.02 & 0.90 \(\pm\) 0.02 & 0.74 \(\pm\) 0.04 & 0.86 \(\pm\) 0.03 \\ GTR-TS-XL & En & 0.83 \(\pm\) 0.01 & 0.77 \(\pm\) 0.01 & 0.80 \(\pm\) 0.01 & 0.36 \(\pm\) 0.01 & 0.83 \(\pm\) 0.02 & 0.70 \(\pm\) 0.03 & 0.87 \(\pm\) 0.02 & 0.91 \(\pm\) 0.02 & 0.73 \(\pm\) 0.04 & 0.86 \(\pm\) 0.03 \\ MPNet-Base & En & 0.75 \(\pm\) 0.01 & 0.68 \(\pm\) 0.01 & 0.75 \(\pm\) 0.02 & 0.81 \(\pm\) 0.02 & 0.79 \(\pm\) 0.02 & 0.57 \(\pm\) 0.03 & 0.80 \(\pm\) 0.03 & 0.88 \(\pm\) 0.03 & 0.70 \(\pm\) 0.04 & 0.80 \(\pm\) 0.04 \\ MSMACO-BERT-Base & En & 0.78 \(\pm\) 0.01 & 0.63 \(\pm\) 0.01 & 0.75 \(\pm\) 0.01 & 0.82 \(\pm\) 0.02 & 0.78 \(\pm\) 0.02 & 0.59 \(\pm\) 0.03 & 0.83 \(\pm\) 0.02 & 0.87 \(\pm\) 0.03 & 0.66 \(\pm\) 0.04 & 0.84 \(\pm\) 0.03 \\ MiniM-L12 & En & 0.78 \(\pm\) 0.01 & 0.70 \(\pm\) 0.01 & 0.71 \(\pm\) 0.01 & 0.84 \(\pm\) 0.02 & 0.78 \(\pm\) 0.02 & 0.64 \(\pm\) 0.03 & 0.84 \(\pm\) 0.02 & 0.89 \(\pm\) 0.03 & 0.72 \(\pm\) 0.04 & 0.83 \(\pm\) 0.03 \\ MultiQM-MPNet-Base & En & 0.79 \(\pm\) 0.01 & 0.70 \(\pm\) 0.01 & 0.71 \(\pm\) 0.01 & 0.84 \(\pm\) 0.02 & 0.64 \(\pm\) 0.03 & 0.82 \(\pm\) 0.03 & 0.82 \(\pm\) 0.03 & 0.90 \(\pm\) 0.02 & 0.01 \(\pm\) 0.04 & 0.84 \(\pm\) 0.03 \\ SGPTP-12GM & En & 0.54 \(\pm\) 0.01 & 0.40 \(\pm\) 0.01 & 0.52 \(\pm\) 0.02 & 0.60 \(\pm\) 0.02 & 0.55 \(\pm\) 0.03 & 0.70 \(\pm\) 0.03 & 0.82 \(\pm\) 0.03 & 0.52 \(\pm\) 0.04 & 0.71 \(\pm\) 0.04 \\ SGPT-Z 7B & En & 0.77 \(\pm\) 0.01 & 0.65 \(\pm\) 0.01 & 0.72 \(\pm\) 0.01 & 0.79 \(\pm\) 0.02 & 0.79 \(\pm\) 0.02 & 0.60 \(\pm\) 0.03 & 0.82 \(\pm\) 0.03 & 0.81 \(\pm\) 0.02 & 0.70 \(\pm\) 0.04 & 0.83 \(\pm\) 0.03 \\ Sentence-T5-base & En & 0.71 \(\pm\) 0.01 & 0.66 \(\pm\) 0.01 & 0.68 \(\pm\) 0.02 & 0.73 \(\pm\) 0.02 & 0.77 \(\pm\) 0.02 & 0.43 \(\pm\) 0.03 & 0.76 \(\pm\) 0.03 & 0.89 \(\pm\) 0.03 & 0.59 \(\pm\) 0.04 & 0.83 \(\pm\) 0.03 \\ Semetence-T5-large & En & 0.74 \(\pm\) 0.01 & 0.64 \(\pm\) 0.01 & 0.65 \(\pm\) 0.02 & 0.76 \(\pm\) 0.02 & 0.76 \(\pm\) 0.02 & 0.47 \(\pm\) 0.03 & 0.80 \(\pm\) 0.03 & 0.89 \(\pm\) 0.02 & 0.59 \(\pm\) 0.04 & 0.81 \(\pm\) 0.04 \\ SyntScience-T5-XL & En & 0.77 \(\pm\) 0.01 & 0.68 \(\pm\) 0.01 & 0.70 \(\pm\) 0.01 & 0.80 \(\pm\) 0.02 & 0.79 \(\pm\) 0.02 & 0.53 \(\pm\) 0.03 & 0.83 \(\pm\) 0.03 & 0.89 \(\pm\) 0.03 & 0.68 \(\pm\) 0.04 & 0.79 \(\pm\) 0.04 \\ Distilled-Base-Multilingual & Eq. 0.69 \(\pm\) 0.01 & 0.57 \(\pm\) 0.01 & 0.58 \(\pm\) 0.02 & 0.75 \(\pm\) 0.02 & 0.74 \(\pm\) 0.02 & 0.53 \(\pm\) 0.03 & 0.80 \(\pm\) 0.03 & 0.72 \(\pm\) 0.04 & 0.75 \(\pm\) 0.04 & 0.74 \(\pm\) 0.04 \\ LaBSE & En & 0.56 \(\pm\) 0.01 & 0.44 \(\pm\) 0.01 & 0.53 \(\pm\) 0.02 & 0.69 \(\pm\) 0.02 & 0.59 \(\pm\) 0.03 & 0.34 \(\pm\) 0.03 & 0.68 \(\pm\) 0.03 & 0.82 \(\pm\) 0.03 & 0.44 \(\pm\) 0.04 & 0.72 \(\pm\) 0.04 \\ LabsSE & Op & 0.64 \(\pm\)

du Naturopathe Vous soigne de toutes vos maladies a l'aide des plantes naturelles moins chere et plus sure sans effets secondaire._

**Translated text: HEPATITIS B AND Cand very reassuring made this experience and dreamed witnessed !!Hepatitis is just an old memory afterwards!REMEDIES TO TREAT AND ERADI-CATE HEPATIIS B FROM THE BODYHEpatitis B is a viral infection that attacks the liver. The virus is transmitted through blood or during sexual intercourse. Indeed, the only secretions or bodily fluids that can transmit the virus are blood, semen, vaginal secretions, saliva and fluids from a wound.Ingredients An unripe Papaya The ROTS of Female Papaya Fresh female papaya leaves Moringa roots fresh Moringa leaves 4 lemons to be cut in half. coconut rootsPreparationPut them all in the pot, the leaves last. Add water and boil the mixture.ManualDrink 2-3 glasses a day. Add water each time and boil once a day. Follow the treatment for a month. Get examined by a doctor and come back to testify.Good recovery...Love your neighbor by sharing this messageLa Boutique du Naturopathe Treats you to all your illnesses using cheaper and safer natural plants without side effects. Detected languages: fra: 100.0%

**OCR transcripts Original text: PoymeraseVirus del'hepatic BParticle filmenteuseADNAntigeneHBSParticule spherique Translated text: Polymerasevirushepatitis BFilamentous particleDNAAntigenHBSspherical particle Detected languages: fra: 72.4%, lb: 9.2%**

### Example #3

g.3.1 Fact-check ID: 93800 Published at: 2021-12-07 factual.afp.com Claim Original text: Nicolas Maduro se fatografio con una camiseta del candidato chileno Gabriel Boric Translated text: Nicolas Maduro was photographed with a shirt of the Chilean candidate Gabriel Boric Dectected languages: spa: 100.0% Title Original text: El tuit de Maduro con una camiseta del candidato chileno Gabriel Boric es un doble montaje Translated text: Maduro's tweet with a t-shirt of the Chilean candidate Gabriel Boric is a double montage Detected languages: spa: 100.0%

g.3.2 Social Media Post ID: 20617 Published at: Facebook 2021-12-03 Verdicts: Altered photo Main text Original text: Vanos con esso apoyos Gabrielito Translated text: Let's go with those support Gabrielito Chetacion Detected languages: spa: 100.0% OCR transcripts Original text: Nicolas Maduro@Nicolas MaduroPor la patria grande, nuestro totalapoyo desde Venezuela al companeroGabriel Boric.Tik Tok #horicchanta #horiccorrrup Translated text: Nicholas Maduro@NicolasMaduroFor the great country, our totalsupport from Venezuela to the comradeGabriel Boric.tik tok #horicccharuta #horiccorrupt Detected languages: spa: 50.4%, eng: 31.6%, qu: 10.0%

Example #4 G.4.1 Fact-check ID: 26926 Published at: 2022-03-16 checamos.afp.com Claim Original text: As fronteiras da Ucrania nao foram registradas na ONU e nao sao reconhecitdas internacionalmente Translated text: Ukraine's borders have not been registered with the UN and are not internationally recognized Detected languages: por: 100.0% Title Original text: As fronteiras da Ucrania sao reconhecidas e nao e preciso que sejam registradas 

na ONU_

**Translated text: _Ukraine's borders are recognized and do not need to be registered with the UN Detected languages:_ por: 100.0%

#### g.4.2 Social Media Post ID: 8853 Published at: Facebook 2022-02-25 Verdicts: False information.

**Main text Original text:**_E esta hein..??!!... havera contraditorio..??...." O secretario-geral das Nações Unidas afimou que a Ucrannia nao solicita registro de fronteira desde 1991, entao o estado da Ucrania nao existe... E nao sabemos disso!!! 04/07/2014 O secretario-geral da ONU, Ban Ki-moon, fez uma declaracao impressionante, cuja distribuica na midia urcaniana e na Internet esta probibida. O conflito entre os dos paises foi discutido na sessao do Conselho de Seguranca da ONU. A partir disso, chegou-se a seguinte conclusao: A Ucrania nao registra suas fronteiras desde 25/12/1991. A ONU nao registrou as fronteiras da Ucrania como um estado soberano. Portanto, pode-se supor que a Russia nao esta cometendo renhumua violaco de diretinos em relacao a Ucrania. De acordo como Tratado da CEI, o territorio da Ucrania e um distrito administrativo da URSS. Portanto,inguem pode ser culpado pelo separatismo e pela mudanca foryada das fronteiras da Ucrania. Sob a leit internacional, o pats simplesmente nao tem fronteiras officialnet reconhecidas. Para resolver esse problema, a Ucrania precisa concluit a demarcacao das fronteiras com os paises vizinhos e obter o acordo dos paises vizinhos, incluindo a Russia, em sua fronteiras comum. E necessario documentar tudo e assinar tratados com todos os estados vizinhos. A Uni\({}^{a}\)e Europoeia prometeu o seu apoio a Ucrania nesta importante questao e decidiu prestar toda a assistencia tecnica. Mas a Rissia assimar um tratado de fronteira com a Ucrania? Nao, claro que nao Como a Russia e a sucessora legal da URSS (isso e confirmado pelas decisoes dos tribunais internacionais sobre disputas de propriedade entre a ex-URSS e raises esteraseros), as terras em que a Ucrania, a Bielerorrussia e a Novorovossiya estao localizadas pertencem a Rissia, e ninguem tem o direiro de ficcar sem o constimento da Russia para dispor desta area. Basicamente, agora tudo o que a Rusasia precisa razer e declarar que essa area e russae e tunedo o que acontece nessa area e un assume interno da Rassia. Qualquer interferencia sera visata como uma medida contra a Russia. Com base nisso, eles podem amular as eleicoes de 25 de maio de 2014 e fazer o que o povo quiser! De acordo com o Memorando de Budapeste e outros accordos, a Ucrania nao tem fronteiras. O estado da Ucrania nao existe (e runca exitiu!).." Alexandre Panin_

**Translated text:**_And this one hull..??!!...there will be a contradiction..??....." The Secretary-General of the United Nations stated that Ukraine has not applied for border registration since 1991, so the state of Ukraine does not exists.... And we don't know that!! 04/07/2014 The Secretary-General of the UN, Ban Ki-moon, made an impressive statement, whose distribution in the Ukrainian media and on the Internet is prohibited. The conflict between the two countries was discussed at the UN Security Council session. From this, the following conclusion was reached: Ukraine has not registered its borders since 12/25/1991. The UN has not registered Ukraine's borders as a sovereign state. Therefore, it can be assumed that Russia is not committing any rights violations in relation to Ukraine. According to the CIS Treaty, the territory of Ukraine is an administrative district of USSR. Therefore, no one can be blamed for separatism and the forced change of Ukraine's borders. Under international law, the country simply has no officially recognized borders. To solve this problem, Ukraine needs to complete the demarcation of borders with neighboring countries and get the agreement of neighboring countries, including Russia, on their common border. It is necessary to document everything and sign treaties with all neighboring states. The European Union pledged its support to Ukraine on this important issue and decided to provide full technical assistance. But will Russia sign a border treaty with Ukraine? No of course not Since Russia is the legal successor of the USSR (this is confirmed by the decisions of international courts on property disputes between the former USSR and foreign countries), the lands on which Ukraine, Belarus and Novorovosisya are located belong to Russia, and no one has the right to be without Russia's consent to dispose of this area. Basically, now all Russia has to do is declare that this area is Russian and that everything that happens in this 

area is an internal Russian affair. Any interference will be seen as a measure against Russia. Based on that, they can nullify the May 25, 2014 elections and do whatever the people want! According to the Budapest Memorandum and other agreements, Ukraine has no borders. The state of Ukraine does not exist (and never did!).." Alexandre Panin_

**Detected languages:** por: 100.0%

### Example #5

#### g.5.1 Fact-check ID: 61827

**Published at:** 2019-11-26 periksafakta.aftp.com

**Claim Original text:**_Gadis gembala di Maroko menjadi menteri penditdikan Prancis setelah dewas_

**Translated text:**_Shepherd girl in Morocco becomes French education minister as an adult_

**Detected languages:** msa: 100.0%

**Title Original text:**_Ini adalah foto seorgan aank Maroko, bukam mantan menteri pendidikan Prancis_

**Translated text:**_This is a photo of a Morocccan child, not a former French education minister_

**Detected languages:** msa: 100.0%

#### g.5.2 Social Media Post LD: 10815

**Published at:** Facebook

**Verdicts:** None

**Main textOriginal text:**_Gadis yg disebelah kiri mengiring domba di maroko, wainita yg discebelah kananadalah gadis vg sama 20 thn kemudian sbg mentri pendidikan pracnis. Jgn pernah berhenti berminnipdan tdk pernnh berhenti bekerja keras utk impian anda.... *** Enerjik. Itulah gambaran sosok Najat Vallaud-Belkacem. Dulunya, dia memakai baju sedanya dengan rambut Mikucir eok kuda, membawa tongkat, dan menggembalakan domba. Sehari-hai di adalah sevang gadis gembala di sebuhah desa lecil diekt Nador, Maroko. Saat iu tidak ada yang menduga balwa kehidupannya ketika dewasa akan berubah jauh lebiu baik. Menjadi menteri pendidikan dan penellitian Pracis. Tentu saja posisi itu tidak begitu saja datang dari langu. Belkacem benvaha edskraeras untub meraihyva. Di kamusuya, takada yang tidak bisa divujadkan. Dulu, ketika indigin berkuliah di Paris Institute of Political Studies, giruk sekolahnya melarangnya meddafar. Alasunya, sekolabit tu mahal sekaligus susah untuk dimasaki. Namun, langkah anak bedua di antara tujuh bersaudara tersebut tak surut. Belkacem tetap mendafar, belajar mati-matian, dan akhirnova diterima. Dia juga harus bekerja pao waktu di dua tempat nutuk membayar biaya kuliadhya. Di kampus iiu pula, dia bertemugan Boris Vallaud yang kini menjadi salah serorange penalishat Presiden Francis Francois Hollande. Meereka sama-sama aktif di Partai Sosialis. Keduany meninkah pada 27 Agustus 2005. Jauhe selbaum itu, Belkacem juga suadh terbias hadtp karas. Saat berusia empat tahn, ayahnux memboyong dia, ibu, dan kakak tertuanya, Fatiha, ke Amiens, kawasan pinggiran Prancis. "Ayah saya tak punya maslah. Tapi, kami, saya, ibu, dan kakak, mati-matian beradaptasi dengan kehidupan baru," kataruya sepetti dikittip Vogue. Dia bahkan sempat terhera-heran saat melihat mobil. Hal tangka di negara asadhya. Belum lagi diskriminasi yang datang dari Lingkungan seki tamrayn. Bahkan saat dia suchah menjadi anggota parlemeni di Rhone-Alpes. Dalam sebual tullisan, Belkacem bercerita, waktu iu dirinya mengadakan perjaminan makan malam dan mengundang tamu yang belum terlalu mengenalnva. Ketika tamu iu datang, Belkacem menyambut dan membantunya melepaskan mantel. Tamu iu lattas bertanya di mana sang permilik rumah. "Hingga saat iii di Prancis, kalau ada preempuan densan kulti hervarna yang membuka piratu ruman di kawasan mewahl, selalu dinaggap penchantu," tulis ibu si kembar Louis-Adel Vallaud dan Nour-Chloe Vallaud tersebut. Sejak saat iu, dia semakinmantap magnabdikan hindup mutku menghilangkan diskriminasi. Sorotan terhadap karir gemilang Belkacem mulai terjadi saat President Francois Hollande menujukhya sebagai juru bicara peemerinah dan menteri hak-hak perenppuan pada 16 Mei 2012. Beberapa bulan setelah iu, Hollande memberiya tangwang jawab untuk memergani homofobia. Belkacem menjabdat menteri pendidikan dan peneliitan pada 25 Agustus 2014, dua hariebeblum ulang lahun kesembolian permikahannya. Penunjukan iut menjadikan da sebagai menteri pendidikan termuda yang pernah dipunvyai Prancis. Terplinihyna Belkacem seakan menjadi budki 

balwa seorang imigran juga bisa menjadi aser yang berharga bagi negara. Apalagi dia adalah soerang muslim. Tentang Belkacem Saat masih kanak-kanak, momen terbaik dalam hidupya adalah ketika bibliobus (mobil perpustakaan kelling) menyambangi kawasam tempat tinggalnya. Sebab, dia bisa membaca bergam buku. Memiliki dua kewarganegaraan. Salah satunya Maroko karena dien kerasal dari sana. Selain itu, Prancis memberinya status wurga negara saat masih lulia. Ia addalu Anak ledua dari tujuh persaudara, Najat Belkacem lahir die negara Maroko padan 1977 di Bni Chiker, sebuah desa dekat Nador di wilayah Rif. Pada 1982 ia bergabung kembali dengan ayahnya, seorang pekerja banguman, dengan ibunya dan kakakuya Fatiha, dan tumbudi subperkotaan Amiens.[3] Ia lulus dari Institut d'etudes politiques de Paris (Institut Studi-Studi Politick Paris) pada 2002. Di Institui ta bertemu Boris Vallaud, yang menikah dengannya pada 27 Agustus 2005.[4] Ia masuk Partai Sosialis pada 2002 dan bergabung densan tim Gerard Collomb, Walikota Lyon, pada 2003 untuk menjalankan demokrasi lokal yang kaut, perlawanan melawan diskriminasi, mempromosilkan hak-hak warga sipli, dan akses mutuk eekrajean dan perumahan. Terpilih dalam Dewan Wilayah Rhone-Alpes pada 2004, ia mengetuai Komisi Budaya, mensundurkan diri pada 2008. Pada 2005, ia menjadi penasihat Partai Sosialis. Pada 2005 dan 2006 ia menjadi kolumnis program kebudayaan C'est tout vu di Tele Lyon Municipale bestama dengan Stephane Cayrol. Pada Februari 2007 iagragabung dalam tim kampanye Segolene Royal sebagai jurbucara, bersama dengan Vincent Peillon dan Arnaud Montebourg. Pada Mater 2008 iater[ilih menjadi conseillere generale departemen Rhone dalam premiibhan Kantonal dengan 58.52% suara pada putaran kedua, dibawah spanduk Partai Sosialis da kanton Lyon-XIII. Pada 16 Mei 2012, ia dilinutik pada kabinet President Perancis Francois Hollandande sebagai Menteri Hak-Hak Wunita dan jurbicara pemeritanhan. [https://m.facebook.com/story.php..._](https://m.facebook.com/story.php..._)

**Translated text:**_The girl on the left is herding sheep in Morocco, the woman on the right is the same girl 20 years later as the French Minister of Education. Never stop dreaming and never stop working hard for your dreams.... *** Energetic. That is the picture of Najat Vallaud-Belkacem. In the past, he wore modest clothes with his hair in a portotail, carried a stick, and herded sheep. Everyday she is a shepherd girl in a small village near Nador, Morocco. At that time no one expected that his life as an adult would change much for the better. Became the French minister of education and research. Of course that position didn't just come from the sky. Belkacem tried extra hard to reach it. In his dictionary, there is nothing that cannot be realized. In the past, when he wanted to study at the Paris Institute of Political Studies, his school teacher forbade him to enroll. The reason, the school is expensive and difficult to enter. However, the step of the second child among the seven siblings did not subsidie. Belkacem continued to apply, studied hard, and was finally accepted. He also had to work part-time at two places to pay for his tuition. On the same campus, he met Boris Vallaud, who is now an adviser to French President Francois Hollande. They are both active in the Socialist Party. The two were married on August 27, 2005. Long before that, Belkacem was also used to living hard. When he was four years old, his father took him, his mother and eldest sister, Fatiha, to Amiens, a suburb of France. "My father had no problems. But, we, me, mother and brother, are desperately adapting to a new life," he was quoted as saying by Vogue. He even had time to be surprised when he saw the car. A rare thing in their home country. Not to mention the discrimination that comes from the surrounding environment. Even when he was already a member of parliament in the Rhone-Alpes. In an article, Belkacem recounted that at that time he held a dinner banquet and invited guests who did not know him well. When the guest arrived, Belkacem greeted him and helped him take off his coat. The guest then asked where the owner of the house was. "Until now in France, if a woman of color opened the door to a house in a luxury area, it was always considered a maid," wrote the mother of twins Louis-Adel Vallaud and Nour-Chloe Vallaud. Since then, he has been steadily devoting his life to eliminating discrimination. The spotlight on Belkacem's illustrious career began when President Francois Hollande appointed him as government spokesman and minister for women's rights on 16 May 2012. Months after that, Hollande gave him the responsibility to fight homophobia. Belkacem took office as minister of education and research on August 25, 2014, two days before her ninth wedding anniversary. The 

appointment makes him the youngest education minister France has ever had. The election of Belkacem seems to be proof that an immigrant can also be a valuable asset for the country. Moreover, he is a Muslim. About Belkacem When he was a child, the best moment in his life was when a bibliobus (mobile library car) visited the area where he lived. Because, he can read a variety of books. Have dual citizenship. One of them is Morocco because he is from there. In addition, France gave him the status of a citizen while still in college. The second of seven children, Najat Belkacem was born in Morocco in 1977 in Bni Chiker, a village near Nador in the Rif region. In 1982 he rejoined his father, a construction worker, with his mother and sister Fatiha, and grew up in the suburb of Amiens.[3] He graduated from the Institut d'Etudes politiques de Paris (Paris Institute of Political Studies) in 2002. At the Institute he met Boris Vallaud, whom he married on 27 August 2005.[4] He joined the Socialist Party in 2002 and joined the team of Gerard Collomb, Mayor of Lyon, in 2003 to promote strong local democracy, fight against discrimination, promote civil rights, and access to jobs and housing. Elected to the Rhone-Alpes County Council in 2004, he chaired the Culture Commission, resigning in 2008. In 2005, he became an adviser to the Socialist Party. In 2005 and 2006 he was columnist for the cultural program C'est tout vu at Tele Lyon Municipale together with Stephane Cayrol. In February 2007 he joined the Segolene Royal campaign team as a spokesperson, along with Vincent Peillon and Arnaud Montebourg. In March 2008 he was elected conseillere generale of the Rhne department in cantonal elections with 58.52% of the vote in the second round, under the banner of the Socialist Party in the Canton of Lyon-XIII. On 16 May 2012, she was appointed to the cabinet of French President Francois Hollanda as Minister of Women's Rights and spokesperson for the government. [https://m.facebook.com/story.php](https://m.facebook.com/story.php)...**Detected languages:** msa: 100.0%