dataesr/scientific-paragraphs-categorization huggingface dataset:
Some systems performed data augmentation and different data cleaning and pre-processing steps. The most successful approaches relied on transformers, achieving up to 0. 956 MAP@5 on the 2022 English tweets dataset. We describe the systems in more detail in Section 4. Several systems used the CheckThat Lab! benchmark without participating in the challenges. Building on the winner of the 2021 competition,  [30]  proposed a framework using distant supervision and a modified self-adaptive training approach to retrieve fact-checked claims for claims made in tweets. Their silver standard corpus is created by harvesting tweets that contain a link to a fact-check from Snopes, similar to  [28] 's mining of tweets and fact-checking articles, yet focused on textual claims and implemented an automatic filtering and labeling step.
They beat the state-of-the-art performance on the CLEF CheckThat! Lab 2021 English tweet data (0. 882) with a MAP@5 score of 0. 903. The web-based online claim retrieval tool ClaimLinker  [12]  links arbitrary texts to claims in ClaimsKG. It obtains candidate claims through a blocking step based on Elasticsearch's (ES) BM25 scores and re-ranks them using textual similarity measures. Despite its simplicity, it succeeds in retrieving the correct verified claim in the first position (P@1) in 76% of the cases for the CheckThat! Lab tweets dataset of 2020 (83. 5% for P@5). MAP scores are not specified by the authors. [31]  preprocess tweets and enrich them with extracted information from embedded URLs, images, and videos. They experiment with different claim numbers for a BM25-based blocking step and different BERT variants for reranking the results.
Their best approaches beat the previous state-ofthe-art performance on CheckThat! Lab English tweet data of the years 2020 and 2021 with MAP@5 scores of 0. 955 and 0. 936, respectively. However, the 2021 system has access to additional metadata. Using the same system as for 2020, the score on 2021 drops to 0. 929. This shows that the datasets of the different years have different characteristics or task difficulties, which makes the different systems proposed in the literature hard to compare, even when they use the CheckThat! Lab benchmark and the same evaluation metrics. We will provide more insights both into the differences of the datasets and the performances of the systems in Sections 5 and 6.
CheckThat! Lab claim retrieval data analyses Several works investigate characteristics of claim retrieval benchmark datasets that are relevant to assess task difficulty, robustness of methods, and possible biases during evaluation. [26] include an exploratory analysis of their dataset, giving examples of trivial cases where the verified claim is identical to the input claim, harder cases where they differ regarding lexical choice, and complex input claims with subclaims, which are verified by two corresponding verified claims. They further distinguish input claim and verified claim pairs into two categories: those that can be matched using simple approximate string matching techniques (Type-1), and all others (Type-2). Analyzing a manually annotated sample of 100 pairs extracted from their politifact debates dataset, they found 48% to be of Type-2.
An analysis of word-level TF. IDF-weighted cosine-similarities further revealed their politifact debates data to be harder than their tweet data from Snopes, with 27% of the debates pairs having similarity scores above 0. 25 vs. 50% of the tweets data. [3]  analyzed the debates dataset by  [26]  in more detail and categorized the pairs of matching claims into 4 categories: (1) clean, (2) clean-hard, (3) part-of and (4) context-dep. (  1 ) and (  2 ) refer to self-contained input claims with a matching verified claim that directly (1) vs. indirectly (2) verifies it, (3) and (4) refer to input claims that are not self-contained and require additional sentences from the discoursive context to fully form an individual claim (3) or need co-reference resolution to contain all relevant information (4).
Furthermore, the authors show that different data splits can greatly influence task difficulty because systems can benefit from training and testing on claims drawn from the same debates or from debates about similar topics. While  [26] 's data and annotation protocol later formed the basis of the CheckThat! Lab datasets, the latter contained additional data and different splits, thus the results cannot be transferred to these benchmark datasets. However, they highlight the importance of analyzing the data in more detail to understand and compare model performance. [32] analyzed the number of words and sentences in the different fields for the CheckThat! Lab 2021 data. [30]  provide insights on similarities between input tweets and titles and subtitles of matching verified claims, compare Jaccard vs.
Datasets For comparative evaluation, we use the datasets and corresponding training/test splits as provided by CheckThat! Lab Task 2 (verified claim retrieval ) challenges of the CLEF2020  [25] , CLEF2021  [6]  and CLEF2022  [7]  initiatives in English language. Table  2  provides statistics regarding training and test data for each dataset. The dataset from the first edition (in the following referred to as 2020-tweets) contains claims extracted from tweets as input queries and corresponding verified claims from fact-checking websites as targets. The targets were drawn from the fact-checking platform Snopes, harvested and augmented with data from ClaimsKG  [10]  as described in  [25]  and  [26]. The data from the 2021 and 2022 challenges each contain one dataset with tweets (Task 2A, in the following referred to as 2021-tweets / 2022-tweets), extensions of the previous datasets, and one with a different discourse type: claims from political debates (Task 2B, referred to as 2021-debates / 2022-debates).
Claims from political debates were mainly drawn from the fact-checking portal Politifact using the same protocol as described in  [26]. The ground truth of links between input queries and matching verified claims was constructed using information provided by Politifact overview articles which feature links between claims from political speeches and fact-checks on their portal, and by Snopes, where the review articles often refer to tweets spreading the factchecked claim. For the tweet data, there is always exactly one correct verified claim to retrieve per input query. For claims from political debates, the number of correct matches may be greater than one. A verified claim may be the correct target for more than one input query in all datasets. Each dataset contains a set of target verified claims to match all queries against: queries from test, training, and development sets alike.
Retrieval performance and robustness For investigating RQ2, we train and test all available methods on all datasets for a comparative evaluation. For RQ3, we perform a cross-dataset evaluation: we apply the trained models to test data of different datasets. This serves 1) to We rely on the performance metrics used in the official CLEF CheckThat! Lab challenges: Mean Average Precision@k (MAP@k) and Precision@k with MAP@5 being the primary measure to rank the performance of systems  [7]. Note that these metrics are applicable both for the single and the multiple target case.
Efficiency For investigating RQ4, efficiency, and scalability, we analyze runtime complexity and costs for online computations of the different approaches. Online processing encompasses all steps that depend on the input claims and can only be executed at inference time: pre-processing and feature generation for input claims, comparisons with verified claims, and inference. Offline processing refers to pre-processing of verified claims in the database, e. g. computing and storing their embeddings or other features, and includes all training and fine-tuning steps. In order to assess applicability in real applications, we measure the time required for the offline and online processing steps of the best supervised and the best unsupervised approach in our comparison. We use two different servers: Server 1 with 4x NVidia GeForce RTX2080Ti (11 GB) GPU, 2 x Intel Xeon 2.
1 GHz CPU with 24 available cores and 1. 4 TB RAM, and Server 2 with 1x A6000 GPU (48 GB) and Xeon E5-4667 v3 CPU with 32 available cores and 64 GB RAM. In addition, we examine whether the best unsupervised approach can be run on a laptop. We use a model with an AMD Ryzen 9 6900HS processor with 3. 30 GHz Radeon Graphics and 16 GB RAM. We measure the average time needed to link an input claim to a claim in a database depending on the size of the database, i. e. the number of candidate verified claims. We compare the results for database sizes of 1k, 5k vs. 10k claims respectively. The respective samples were drawn randomly from the 2020-tweets dataset with all claims from the smaller sets being included in the larger ones. We do not store and use any intermediate results such as embeddings of already considered input claims and treat all experiments with the different database sizes independently.
Claim retrieval methods and baselines In the following, we introduce the methods compared in this study: systems and baselines that were part of any of the CLEF CheckThat! Lab claim retrieval challenges (2020, 2021, and 2022) plus the online claim retrieval tool Claim-Linker, and an extension of the 2022 participant SimBa, SimBa 2023. We include here all systems for which publications or working notes are available. An overview of the systems and their characteristics can be found in Table  3. While many systems experiment with the use of different verified claim fields, most use only the texts of the verified claims or the texts plus the titles of the fact-checking articles for prediction. Likewise, some systems experimented with incorporating the context information from the debates transcripts but none used them in their final submissions.
The authors accidentally submitted results generated by averaging the scores instead of using the better-performing meta classifier. We provide the results of their best-performing latter variant (Fraunhofer SIT SVM) in addition to the submitted variant. For all publicly available submissions, detailed in the following, we additionally provide detailed cross-evaluation experiments and replicate their results. We modified the approaches to allow training and testing on all different datasets. All modified code can be found in our repository. Unless stated otherwise, we used the variants that constituted the teams' primary submissions. TIET  [41]  retrieves the top 1,000 claims for an input tweet based on their BM25 scores and combines those with a similarity score based on the cosine between Sentence-BERT (Base NLI Mean Tokens) embeddings for input and candidate claims.
Data Analysis In order to facilitate an informed interpretation of experimental results of stateof-the-art systems and the utilised features, understanding the characteristics of the used datasets is crucial. This is even more important when aiming to understand model robustness and overfitting through cross-dataset evaluation. Thus, in the following, we provide an assessment of the aforementioned widely used benchmark datasets for claim retrieval evaluation. We draw on characteristics investigated and deemed relevant for the task in previous works (cf. Section 4), such as claim and input lengths and token overlaps, and provide extended analyses of these properties for all different datasets, including statistics for inputs and verified claims in test vs. training splits. We add detailed insights into the completeness and correctness of the data and gold data annotations and their match to the task definition and real-world claim retrieval requirements.
Matches per input query and per verified claim As per task definition, each input query in the tweets datasets has one matching verified claim in D. However, we find that in 2020-tweets, one training query has two matching verified claims. In the 2021/2-tweets training set, this is the case for two input queries: the query corresponding to the 2020-tweets query plus a new entry. Moreover, there are duplicates in some of the gold input query -matching target verified claim pair files for the test datasets, as detailed in Table  4. Duplicates here mean repeated entries of the same input query -verified claim pair. Thus, the respective input queries implicitly gain a higher weight during evaluation than the ones that are not repeated. There are no duplicates in the input query files. For 2020-tweets, one test input query (ID 1198) does not have a match to a verified claim in the gold file.
Instead, there is a duplicate for another input query (ID 1167). A closer examination of the data reveals that the duplicated line can be corrected by changing the listed query ID with the missing query ID, i. e. both match the same verified claim (see below for statistics regarding multiple matches per verified claim). All other input queries are contained in the gold files. The other tweet datasets do not contain duplicates. However, as Table  4  shows, there are many duplicates in the debates test sets. For the debates data, 12. 288% of training queries have more than one matching verified claim. For the test queries, this is the case for 16. 456% (29. 114% without de-duplication) for 2021debates and 18. 462% (26. 154% without de-duplication) for 2022-debates. The average number of matching verified claims per test query remained stable: 1.
165 (1. 304 without de-duplication) for 2021-debates vs. 1. 185 (1. 277 without de-duplication) for 2022-debates. For training queries, the ratio is 1. 191 (1. 191 without de-duplication). All following analyses are performed on the original datasets, without de-duplication, to allow insights into the results reported in the literature which used the data as-is. For all datasets, one verified claim may be the correct match for multiple input queries. Table  4  details the distribution for each dataset and split. For all datasets, the number of positive examples is low. The debates datasets contain fewer positive examples than the tweets datasets. When a verified claim is the correct match for an input query, it in many cases also matches other input queries (∼18-20% for tweets training datasets, ∼12-28% for tweets test, ∼36% for debates training, ∼37-44% for debates test).
This is pronounced for the debates data. Therefore, the number of verified claims to choose for matching is low, as detailed in column #c match, and models may learn to identify relevant verified claims and that each of them has a high probability of matching multiple input queries. In eight cases, a verified claim is the correct target for input queries in the training set and input queries in the test set, i. e. a model may learn to find matching claims for this specific verified claim in the training phase. This is true for seven verified claims in 2020-tweets and one claim in 2022-tweets. dataset split #V #c #c match #c.
Tweet datasets For the tweets data, training, and test input queries consist of tweet texts including links to pictures or other URLs, user mentions, hashtags, and all other content allowed in tweets. In addition, they have the user name and mention as well as the date of publication appended to their end, separated from the tweet text by "-". For an example, see Table  5. The query with q id 2 is one of three queries in the 2020-tweets training set that is missing the Table  5 : Matching queries and targets from the 2020-tweets training dataset. appended information. The 2021-tweets training set misses this information in four cases. In all other data, the appended information is present. Therefore, check-worthy claim in this task refers to a tweet, which may consist of multiple sentences and a user mention and publication date.
Political debates datasets For the debates data, the target verified claims consist of verified claim text (vclaim) and metadata from fact-checking review articles: URL, speaker, truth label, date, title and review text (truncated here), as shown in Figures  3  and 4. { "url": "/factchecks/2020/jan/15/elizabeth-warren/democratic-debates-biggestelectoral-losers-number/", "speaker": "Elizabeth Warren", "vclaim": "\"Look at the men on this stage. Collectively, they have lost 10 elections. The only people on this stage who have won every single election that they\u2019ve been in are the women, Amy and me. \"", "truth_label": "true", "date": "stated on January 14, 2020 in a debate:", "title": "The Democratic debates\u2019 biggest (electoral) losers, by the numbers", "text": <full review text>, "vclaim_id": "vclaim-pol-17583" } Fig.
3 : Target verified claim with ID vclaim-pol-17583 from the 2021-debates test dataset { "url": "/factchecks/2016/jan/13/barack-obama/business-has-created-jobsevery-month-obamacare-be/", "speaker": "Barack Obama", "vclaim": "\"Our businesses have created jobs every single month since (Obamacare) became law. \"", "truth_label": "true", "date": "stated on January 12, 2016 in his State of the Union speech:", "title": "Business has created jobs every month since Obamacare became law, Obama said in State of the Union", "text": <full review text>, "vclaim_id": "vclaim-pol-05563" } Fig. 4 : Target verified claim with ID vclaim-pol-05563 from the 2021-debates test dataset Note that the speaker field may contain all kinds of information about a source of a fact-checked claim: it can be the actual speaker of an utterance, as in the example above, or other source information such as "Facebook posts".
The test datasets include transcripts of the speeches, containing not only the text of the input claim utterances but also their discoursive contexts. The input claims themselves are individual sentences from the speeches, yet they sometimes refer to entities mentioned in the context rather than the claims themselves, and can depend on one another, as can be seen in Table  6. With this, the data can be seen as diverging from the task definition defining a check-worthy claim as input, as the individual queries in this dataset are not necessarily check-worthy by themselves and do not constitute individual claims, depending on the precise definition of what a claim is (see  [4]  for an overview of diverging definitions), which is left implicit in this task definition. In both the 2021 and 2022 datasets, the input query IDs contain the date of the debate which can be used as a feature to compare with the verified claim dates (cf.
Word overlap In order to assess the difficulty of linking queries and target claims for the different datasets and the task in general, we measure the ratio of overlapping tokens/lemmas in input queries and targets, for both only the matching targets and all target verified claims including the false matches. For this, we tokenize and lemmatize the input using spacy 37 , count the common tokens/lemmas for each pair of input query and target claim and normalize the count by the number of tokens in the smaller of the two sentences. This is because, as detailed in the task definition of 2020 (see Section 3), the target may be a subclaim of the query or vice versa to justify a match and we do not want to give a higher weight to either of the directions in this analysis. We use the original data for the analysis, i.
e. we do not perform de-duplication, as the original data is what has been used by the systems we strive to evaluate. As few systems use subtitle information, see As Table  7  shows, the input queries and matching verified claims in the debates datasets have a higher token overlap than the ones in the tweets datasets (0. 385 vs. 0. 342 for train and 0. 375 vs. 0. 342 for test on average). For the non-matching pairs, overlaps are more frequent in the debates datasets as well (0. 156 vs. 0. 121 for train and 0. 157 vs. 0. 113 for test on average). Overlaps for matching queries and targets are generally high both in the training sets (mean: 0. 359) and the test sets (mean: 0. 355) for all datasets, while the overlaps between queries and all targets are much lower (train mean: 0. 135, test mean: 0.
130). The 2020-tweets test set has a considerably lower overlap ratio between input queries and verified claims than all other datasets. While the overlaps are lower for the matching pairs as well, the difference between the overlap ratios of matching and all pairs is greater in this dataset, with the matching pairs having roughly 3. 7 times as many overlaps compared to 2. 5 (2021-tweets), 2. 8 (2022-tweets), 2. 3 (2021-debates) and 2. 4 (2022-debates). A similar pattern can be observed for the titles. In contrast to the claim texts, the overlap ratios are higher in the tweets than in the debates datasets. For both discourse types, overlap ratios are higher for claim texts than for claim review titles. While the overlap of lemmas naturally is higher than the overlap of tokens, the ratio of overlaps is already high without any pre-processing.
The performance of BM25-based baselines adds further insights regarding the similarities of input queries and matching verified claims for the different datasets. As these are included in the comparison of approaches in Section 6, we omit these statistics here. Subfigures (a) and (d) of Figure  5  and Table  8  show that the distributions of verified claim lengths are similar in the different tweet datasets, with the mean of tokens ranging between 19. 0 and 19. 6, while debate claims are longer on average  (22. 7) , but vary more strongly with a standard deviation of 9. 6 vs. 7. 4 -7. 5 for tweets. Verified claims matching input training and test queries are longer than the average of all verified claims for 2021-tweets and 2022tweets, while the test input query matches are shorter for 2020-tweets and the debates datasets.
Input query and target verified claim lengths The length of verified target claim titles (cf. Subfigures (c) and (f) of Figure  5  and Table  5 ) fluctuates more strongly across datasets (mean #tokens: 9. 0 for 2020-tweets, 10. 4 for 2021-tweets and 2022-tweets, standard deviation 4. 4 -4. 8 for 2020-tweets) and they are longer in the debates data (mean #tokens: 13. 6, standard deviation 4. 0). Verified claim titles matching training queries are longer than all others for the tweets datasets and for 2021-tweets and 2022tweets, the titles of claims matching test queries are longer than the average, too, but not for 2020-tweets. Similarly, the length of training input claims (cf. Subfigures (b) and (e) of Figure  5  and Table  9 ) is similar for the tweets datasets (mean #tokens: 42. 3, standard deviation 15.
8 -15. 9) with them being complete tweet texts with appended information. Debates training input claims are shorter (mean #tokens: 24. 1, standard deviation 16. 3), as they are individual statements uttered in a speech or debate. A comparison of the last two columns of the figure and the corresponding Table  9  reveals that the distributions regarding input query length differs across training and test splits for all datasets except 2022-debates, most strongly for the 2020-tweets data. Striking is the difference between test and training input queries in 2020-tweets, where the test queries are much shorter than the training queries (29. 8 vs. 42. 2) and have less variance regarding their length (5. 8 vs. 15. 9). Many training input queries and target claims contain a very low or high number of tokens (cf.