dataesr/scientific-paragraphs-categorization huggingface dataset:
Such errors are hard to avoid when automatically extracting information from articles of fact-checking portals, as their structure is not fully consistent and the reviewed claims are not always explicitly listed, see, for instance, the fact-check at https://www. snopes. com/fact-check/citibank/ which is part of 2020-tweets where it is represented as {"target_id": "4187", "target": "Citibank customers. ", "title": "Citibank Phish"}. The short input queries in the debates dataset are due to the fact that they may refer to the context, as outlined above and in Table  6 , e. g. ID 20161019 3pres 0423: "Wrong", and ID 20160303 GOP michigan 0575: "Ten times". This is in line with  [3] 's analysis of  [26] 's base debates dataset, where they found many claims that are not self-contained. With this, they do not represent input that is expected in a real claim retrieval system, as users can typically be expected to enter a claim with its context.
Note that, however, the context is accessible in the transcript files and systems participating in the Check That! Lab challenge may use them to enrich the input queries. When processed without additional context, debates input queries contain much less information (tokens and characters) than the tweets datasets. As Figure  5  reveals, there is a long tail of very long claims and queries. We define outliers as those queries and claims whose token counts exceed the mean +2 * standard deviation. The results of the analysis are listed in Table  10. As Table  10  details, 3-4% of all target verified claims and those matching input training queries in the tweets datasets exceed the maximum expected length. The debates claims are not only longer on average, they also contain  more long outliers.
Summary The CheckThat! Lab claim retrieval datasets feature input queries and targets with many direct lexical We can thus expect that claim retrieval methods relying on simple word overlap metrics can achieve high scores, even without lemmatization or similar pre-processing. Likewise, embedding-based systems can be expected to perform strongly when the vocabulary matches the dataset contents. Overlaps for queries with verified claim texts are higher than with titles and titles on average contain fewer tokens than the verified claims. Thus, titles might be less helpful when used in systems relying on lexical similarities to a large degree, but they could add more focused information with semantically similar concepts, as they usually are abstractive summaries of the claims. Check-worthiness detection and claim retrieval are separate tasks.
According to the task definition, input queries for the claim retrieval task are check-worthy claims or tweets. However, indeed input queries are not always individual, self-contained claims that constitute check-worthy claims in themselves as also visible in  [26] 's analysis on their dataset that formed the basis of the Check That! Lab data. Therefore, models training on this data may not be able to distinguish essential parts of queries and targets, i. e. those that contain the check-worthy information, but only learn to find any matching information. Especially in light of the importance of word overlaps, the length of queries and verified claims can play a large role. Very short, long, or noisy claims can lead to overfitting. We also find erroneous claims in the datasets due to the difficulties of automatic harvesting from semi-structured fact-checking portals that can potentially increase the task difficulty.
However, this mirrors data for real claim retrieval applications. Thus, investigating the robustness of methods regarding noise can yield valuable insights into the applicability of systems for applications. The tweets datasets are similar to one another, and so are the debates datasets. 2020-tweets stands out regarding word overlaps: here, word overlaps can be expected to be a better feature to distinguish matches from non-matches than for other datasets. This dataset also has shorter test queries with low variance regarding their length but high variance regarding the length of the training queries. This dataset contains less information for the verified claims, e. g. no date field. As dates are appended to the input tweets, models extracting this data for matching with the targets are expected to yield better results for 2021 and 2022 tweets data than for 2020.
In all datasets, the IDs give away some information: 1) their ordering (succeeding input queries may link to the same target) and 2) info about dates, events, speakers that can be extracted from them and used for linking (debates). When comparing the performance of different systems, it should be checked whether they make use of such information to gain a more realistic assessment of their performance in real-world systems. In real-world applications, there may be no correct match for an input query while in the given data, there always is one match per input query for the tweets datasets and at least one match per input query for the debates data. the data can be used to develop methods that distinguish relevant from non-relevant claims. While the task is formulated as a ranking task, the gold data distinguishes relevant verified claims from irrelevant ones without ranking multiple valid matches.
Performance on different datasets Tables  19  and 20  detail the results of all approaches for training and testing on data of the same year and discourse type. Tables  14, 15 , 16, 17 and 18 detail the performance by dataset. The MAP@k scores for UofSheffield remain constant for all values of k because this system returns only the top-1 matching verified claim per input query instead of a ranked list. When only considering approaches that have participated in the challenge of a respective year (marked with ⋆), we can see that the top score was always achieved by a supervised method. Adding all approaches under investigation, we find that the unsupervised SimBa 2023 outperforms the best supervised approach for 2020-tweets, BigIR (cf. Table  14 ). The winner of the 2020-tweets competition was Buster.
AI with a score of 0. 929. This approach ranks third in the overall comparison for this dataset. Surprising is the relatively weak performance of RIET Lab, which is the best-performing approach on all other datasets. This is due to one out of three runs where the model failed to learn maximally meaningful features: while the best run scored 0. 957, the worst run achieved 0. 801. For 2020-tweets, but not the other datasets, two index-based approaches yield the top scores regarding precision (BigIR and ES Baseline), but not MAP, which includes recall. Also, the precision scores are generally higher than for the other datasets. The data analysis (Section 5) showed that 2020-tweets features a relatively low token overlap of non-matching input queries and verified claims and a much higher overlap ratio in matching pairs.
Thus, simple token matching techniques serve well to find matches with high precision. At the same time, absolute overlap ratios are lower than in the other tweet datasets, which hints at the presence of many pairs for which semantic similarity measures are required. In line with this, we find that models focusing strongly on embedding-based similarities, such as SimBa 2023 and Check square: distmult, yield both higher precision and MAP scores on 2020tweets than on the other even though the ES Baseline has its highest performance on this dataset, closely followed by 2022-tweets. For 2021-tweets, the winning approach is RIET Lab (0. 956), followed by SimBa 2023 (0. 945), BirIR (0. 936), and the extension of the winner of the 2021 competition, Aschern (0. 883): CrowdChecked (0. 903). For the latest 2022 edition, the best performance on the tweets data is again achieved by RIET Lab with a score of 0.
5 while for the English tweets, the top scores reach ∼0. 96 MAP@5. The difficulty of the 2021-debates and 2022debates test sets is similar, with the latter being easier with MAP@5 scores ranging from 0. 238 -0. 501 vs. 0. 196 -0. 480 for 2021-debates. The ES Baseline performs much lower than on the tweets data, yet the index-based approaches ClaimLinker, TIET, and ES Baseline work relatively well for the debates data compared to other systems. Their scores are generally similar to one another, except for 2020-tweets. As detailed in Section 5, the difference between token/lemma overlap ratios in the negative vs. positive examples of the test set is the highest for this dataset. Thus, text overlap features seem to be particularly beneficial, which may explain ClaimLinker 's performance bonus compared to the ES Baseline.
Across datasets, the performance of the supervised approaches fluctuates considerably, even within the tweet datasets. Striking is the performance drop for UofSheffield for 2021-tweets, and, to a lesser degree, 2022-tweets (cf. Table  20 ), NLP&IR@UNED's drop for 2021-tweets as well as RIET Lab's lower performance and high fluctuation on 2020-tweets. The highest standard deviation for the supervised approaches is obtained by NLP&IR@UNED for 2022-tweets with a maximum MAP@5 score of 0. 506, a minimum of 0. 235 and a standard deviation of 0. 112 for all three runs. Our data analysis in Section 5 revealed many outliers regarding verified claim length. Investigating the predictions in more detail, we find that NLP&IR@UNED has a strong bias of classifying very long claims (i. e. claims with more than 34 and 42 tokens, which are the average outlier ranges for the tweets and debates datasets, respectively, cf.
Section 5. 4. 2) as correct matches for input queries (see Table  12 ). To generate these numbers, we counted for Robust and Efficient Claim Retrieval for Online Fact-Checking Applications each verified claim with excessive length how frequently their corresponding IDs occur at the top and within the top-5 matches of the predictions files. As revealed by Table  12 , the counts vary heavily in between runs for NLP&IR@UNED and the counts are generally high. The other systems do not exhibit a similar bias, regardless of whether they use only vclaims or vclaims plus titles. All systems but UofSheffield rightfully include a smaller number of long claims in their predictions for 2020-tweets than for the other tweet datasets. As outlined in Section 5, 2020-tweets has considerably shorter test input queries with shorter matching verified claims, but the length of the matching training verified claims does not differ from the other tweets datasets.
Instead, a system may be worse at picking up spurious correlations, as explained by  [8]. We find hints for this in our comparison of Check square using fine-tuned BERT vs. pre-trained DistilBERT embeddings and the greater preference of the latter for choosing long claims as matches for 2020-tweets. A related observation was made by  [39]  who found that fine-tuning using titles did not improve their system, but improved its robustness. In general, using the more abstractive titles can intuitively be expected to help a system deal with paraphrased input claims, and many systems use them in addition to the vclaim texts, cf. Table  3. However, the methodology of dataset collection might have introduced biases, e. g. fact-checkers listing tweets or statements they checked in their articles might be likely to phrase their summarizing claims in a similar way.
In this case, the importance of abstractive titles as an additional information source might be pronounced in real-world data. However, the correlations present in the data may also reflect properties of real-world data, i. e. input queries and matching verified claims naturally share many tokens, such as entities mentioned in their most common form. As the different behavior of the systems is not always visible using all of the datasets, this analysis highlights that testing on the different dataset is important to assess model performance and biases for this task. Our results generally verify the observations made in the literature that the choice of language models for this task greatly impacts the results: For Check square, the authors found on the 2021-tweets data that multilingual DistilBERT embeddings without fine-tuning outperformed fine-tuned monolingual Sentence-BERT models.
For the former, fine-tuning using their triplet loss methodology hurts performance while it helps in the case of Sentence-BERT  [45]. Similarly,  [39]  found on the same dataset that their RoBERTa model fine-tuned on triplets performed worse than monolingual DistilBERT without fine-tuning. We could replicate that Check square's variant using multilingual DistilBERT embeddings without fine-tuning outperforms the fine-tuned BERT embeddings on all datasets but 2022-tweets. For this dataset, both variants perform on par. Similarly, according to  [26] , Sentence-BERT and Universal Sentence Encoder performed much better in their system and on their data than BERT, and Sentence-BERT better than Universal Sentence Encoder. In the same vain,  [34]  found on 2020-tweets that using a standard BERT model to represent sentences resulted in low performance while pre-trained Sentence-BERT performed a lot higher and further benefited from fine-tuning.
? Method P@1 P@3 P@5 MAP@1 MAP@3 MAP@5 Table  14 : Results for 2020-tweets. Sup. ? = Supervised/fine-tuned system? Numbers for approaches marked with * have been taken from the respective publications. Scores not detailed therein are marked with * in the respective columns. All other numbers we generated/replicated for this study. Systems marked with ⋆ participated in the challenge of that year very large language models while SimBa 2023 employs high-performing language models in their pre-trained base variants without any supervision. For SimBa 2023 and its choice of language models combined with lexical features, we find no disadvantage regarding model robustness. Finally, we repeated the analysis using the de-duplicated gold files. While the scores deviate for datasets with duplicates, 2021-debates and 2022-debates, cf.
? Method P@1 P@3 P@5 MAP@1 MAP@3 MAP@5 Numbers for approaches marked with * have been taken from the respective publications. Scores not detailed therein are marked with * in the respective columns. All other numbers we generated/replicated for this study. Boldface marks the best scores on a dataset for the supervised approaches, underlined scores are the best scores for all approaches, including the unsupervised ones (cf. Table  19 ). Blue color marks the best scores of an approach across datasets.
Cross-dataset evaluation The cross-evaluation results are displayed in Tables  21  and 22. Since we reported averaged results in the previous subsection, the performance of a trained model on the test dataset of the same type and year does not correspond to the values in Table  20. We chose a random run (e. g. always the model trained in the last run) rather than selecting a specific model. As the results show, training an approach on a dataset of a different year or type does not necessarily hurt performance. Instead, we find that models that perform well on one dataset perform well on others, too. E. g. NLP&IR@UNED and Uof-Sheffield learnt their most useful model on 2020-tweets. To control for biases introduced by the duplicates in the gold files, we repeated the evaluation with the de-duplicated versions.
Method Dataset (training → test) P@1 P@3 P@5 MAP@1 MAP@3 MAP@5    22 ), underline the best score across all datasets and approaches, blue color the best scores for an approach for each test dataset. Note that using the Sentence Transformer, embeddings are projected to Euclidean space which enables search with O(i+D) inferences with i being the number of input claims, instead of O(i * D) pair-wise comparisons (cf. [46] ). For real applications, especially web-based ones, the magnitudes of the constants matter. As detailed in Subsections 6. 1 and 6. 2, RIET Lab is the approach with the highest performance on most datasets but has very high online and offline    21 ), underline the best score across all datasets and approaches, blue color the best scores for an approach for each test dataset.
computational costs for training and fine-tuning and requires the availability of a high-performance GPU server. SimBa 2023 achieves competitive scores, and its offline computations are relatively cheap, consisting only of computing embeddings using pre-trained language models and other textual features for the verified claims. In order to answer RQ5, we now investigate the question whether the two best performing approaches, supervised RIET Lab and unsupervised SimBa 2023, are suitable for online fact-checking applications where response times matter. We experimentally measure their offline computation and online retrieval times. Figure  8  shows the time required to link the set of 200 claims. Table  24  details the average time required to retrieve verified claims for one input claim, depending on dataset size, illustrating both total runtimes and scalability.
The offline computations for RIET Lab consist of training of the transformer-based candidate selection blocking step, and fine-tuning the GPT-Neo 1. 3 Billion parameter model as reranker. For SimBa 2023, offline computations refer to the generation of three different embeddings plus tokenization for all verified claims. As shown in Figure  8a  and Table  24 , the transformer-based blocking step in RIET Lab is more efficient than SimBa 2023 's pair-wise comparisons regarding complexity, but RIET Lab's constant costs are more than 200 times higher on Server1 and ∼10 times higher on Server2 for a database size of 10k. The differences regarding the runtimes for Server1 vs. Server2 are due to the fact that RIET Lab benefits greatly from   GPUs, but the GPUs on Server1 do not meet the requirements regarding available memory and thus the computations were performed on CPUs only.
Discussion To answer RQ1, our data analysis in Section 5 reveals that the benchmark datasets for the claim retrieval task are characterized by a high rate of token overlaps, which explains the relatively high performance of baseline methods relying on BM25 and other lexical similarity scores, as well as the dominance of embedding-based methods (cf. Section 6). As the benchmarks are built using real data, they contain noise, such as very long and very short claims, partly due to extraction errors. As this reveals vulnerabilities of methods regarding overfitting and allows to better measure their robustness, we argue that this actually makes the data more useful to assess the suitability of methods in real-world systems, where they will be suspected to this kind of noise as well. Data from different discourse types, tweets, and political debates, are similar regarding characteristics such as ratio of token overlaps.
The increased task difficulty for the debates data can to a large part be attributed to the lacking context information in its input queries, a challenge that would typically not arise for our use-case, a real-world claim retrieval system with queries entered by users. Therefore, these different datasets are not very well suited to investigate the general generalizability of models across different data distributions. The tweets datasets are limited in the sense that there is always one matching verified claim for each input query, while in real systems drawing on multiple fact-checking portals, we can expect anything from zero to multiple potential matching verified claims, since different fact-checking portals may check the same check-worthy claims. The debates data loosens these restrictions, allowing multiple matching targets.
Investigating RQ2, we find that the highest performance is achieved by supervised methods on all datasets but 2020-tweets (cf. Tables  19  and 20 ). The best-performing approach overall is transformer-and GPTNeo-based RIET Lab, which achieves up to 0. 956 MAP@5 on 2022-tweets, followed by the unsupervised SimBa 2023, which achieves the top score on 2020-tweets with 0. 962. While supervised/fine-tuned systems also generally dominated the leaderboards of the CLEF CheckThat! Lab challenges, their performance turned out to fluctuate considerably between and within different datasets and is hard to predict, e. g. with RIET Lab performing ∼0. 1 points worse on 2020-tweets than on the other tweet datasets, with individual runs ranging from 0. 801 to 0. 957 in MAP@5 and UofSheffield 's performance diverging by ∼0.
5 points between their 2020-tweets and 2021-tweets models (cf. Tables  19 ), despite the similarity of the datasets (cf. Section 4). As our cross-evaluation results in Tables  21  and 22  show, this difference is caused by what the model learned rather than by differences in the test sets. This and our analyses showing that noisy data may greatly bias the systems' performance (cf. Tables  12  and 13 ) highlight the importance of the availability of large amounts of training data to avoid overfitting, especially in the presence of noisy data. Acquiring such data, however, is costly, and so are training and fine-tuning using large language models. Unsupervised methods reach a similar or even better performance, with SimBa 2023 achieving the top rank on the 2020-tweets data with a MAP@5 score of 0.
964 and most unsupervised methods outperforming most supervised ones, except RIET Lab, on the debates datasets (cf. Tables  17  and 18 ). SimBa 2023 is the best unsupervised method, ranking second behind RIET Lab on all datasets except 2020-tweets. On this dataset, it achieves the top score (cf. Tables 14, 15, 16, 17, 18). Regarding RQ3, our results show that supervised methods for this task generalize well across the used datasets of different discourse types, i. e. models learned on tweets data work well for retrieving claims from debates and vice versa. However, as the models are prone to overfitting, their performance depends heavily on the training data and fluctuates strongly within the same discourse type (cf. Tables  21  and Tables 22 ): E. g. NLP&IR@UNED and Uof-Sheffield learned their most useful model on 2020-tweets which then yields the best results on all datasets, including those featuring claims from political debates.
In general, training models on the 2020-tweets dataset yields the bestperforming models for most systems. This is the dataset where token overlaps are the most indicative to distinguish matching and non-matching pairs, the variance in input query length is highest in the training set and lowest in the test set. As the datasets are relatively small, however, supervised methods may overfit their models: e. g. RIET Lab's worst and best model trained on 2020tweets yields a MAP@5 score of 0. 801 vs. 0. 957, respectively. The debates data features fewer positive training examples than the tweets data, shorter inputs (i. e. less information), and the imbalance of negative vs. positive examples is pronounced. Training on this data yields the lowest performance. Finally, the cross-evaluation confirms that not only the training data for the debates data is more difficult, but also the test sets are harder than those of the tweets datasets: models trained on the tweets data that perform well across datasets still yield lower scores on the debates test datasets.
These have a higher number of non-matching verified claims to choose from, multiple correct matches, the input queries contain less words and characters than the tweet input queries and are less self-contained (cf. Section 5). Section 6. 3, Figure  8  and Table  24  answer RQ4: methods relying on an index structure naturally exhibit constant low retrieval time. The use of blocking techniques can render approaches using costly computations competitive in regards to efficiency without yielding bad performance (cf. Table  19 ). Efficiency can be greatly improved by performing costly computations for all claims in the dataset offline, storing them for comparisons with new input claims, and computing the costly features only for the latter. In combination with blocking, this can cap and reduce the retrieval time of approaches making heavy use of costly features.