AuRED: Enabling Arabic Rumor Verification using Evidence from Authorities over Twitter (2024)

Abstract: Diverging from the trend of the previous rumor verification studies, we introduce the new task of rumor verification using evidence that are exclusively captured from authorities, i.e., entities holding the right and knowledge to verify corresponding information. To enable research on this task for Arabic low-resourced language, we construct and release the first Authority-Rumor-Evidence Dataset (AuRED). The dataset comprises 160 rumors expressed in tweets and 692 Twitter timelines of authorities containing about 34k tweets. Additionally, we explore how existing evidence retrieval and claim verification models for fact-checking perform on our task under both the cross-lingual zero-shot and in-domain fine-tuning setups. Our experiments show that although evidence retrieval models perform relatively well on the task establishing strong baselines, there is still a big room for improvement. However, existing claim verification models perform poorly on the task no matter how good the retrieval performance is. The results also show that stance detection can be useful for evidence retrieval. Moreover, existing fact-checking datasets showed a potential in transfer learning to our task, however, further investigation using different datasets and setups is required.

heading: 1 Introduction
text: The spread of rumors and fake news on social media causes anxiety and panic in communities, forming persistent challenges for platforms, policymakers, and researchers. To address this, several rumor verification studies on social media incorporate the propagation networks as a key source of evidence. They either utilize the stance of replies (Kumar and Carley, 2019; Yu et al., 2020; Bai et al., 2023), the structure of the replies (Ma et al., 2018; Bian et al., 2020; Song et al., 2021), or the users’ metadata (Liu and Wu, 2018). On the other hand, evidence are extracted from the Web to augment signals from
the propagation networks (Dougrez-Lewis et al., 2022; Hu et al., 2023). However, studies on Arabic rumor verification incorporating evidence are scarce. Haouari et al. (2021) and Althabiti et al. (2022) exploited the tweet replies, while Albalawi et al. (2023) leveraged the images and videos embedded in the rumor tweet.
Authorities (i.e., entities having the real knowledge or power to verify or deny a specific rumor (Haouari et al., 2023; Haouari and Elsayed, 2023)) can also be a valuable source of evidence that augments other sources for verifying rumors, either by automated verification systems or more specifically by human fact-checkers. Detecting the stance of authorities towards rumors in Twitter1 was indeed introduced recently as a potential signal for better rumor verification (Haouari and Elsayed, 2023, 2024). However, to the best of our knowledge, no study to date has explored the incorporation of evidence tweets retrieved from the timelines of authorities for rumor verification over social media in general and for Arabic rumor verification in particular. Additionally, there is no available dataset for that task to support such research.
To bridge this gap, in this paper, we introduce the problem of rumor verification using evidence from authorities over Twitter and a dataset that enables research tackling that problem. The problem is defined as follows: given a rumor expressed in a tweet and a set of Twitter accounts of authorities for that rumor, the system should retrieve evidence tweets posted by any of those authorities. Based on the retrieved evidence, the system should determine if the rumor is supported, refuted, or unverifiable. Figure 1 illustrates the setup of the problem.
To facilitate the research on this task, we introduce AuRED, the first Authority-Rumor-Evidence Dataset. AuRED covers 160 Arabic rumors an-
1“Twitter” is the former name of “X,” however we will use “Twitter” for clarity.
27
notated with tweet-level evidence from their corresponding 692 authority timelines, comprising about 34k annotated tweets in total. The dataset was constructed by annotating a set of rumors, selected from two existing datasets (Haouari et al., 2023; Haouari and Elsayed, 2024), following two main steps (1) finding authorities that can help verify the rumors (Haouari et al., 2023), and (2) collecting the timelines of those authorities, and annotating those timelines to find evidence tweets.
Our contribution in this work is five-fold:
• We propose the new task of rumor verification using evidence from authorities over Twitter. • We introduce AuRED,2 the first Arabic public dataset for the task. • We present benchmarking results on AuRED, and release our source code for reproducibility and facilitating research on the task.3 • We explore how existing evidence retrieval and claim verification models, that are originally proposed for fact-checking, perform on our task under both the cross-lingual zero-shot and in-domain fine-tuning setups. • We investigate the usefulness of detecting stance of authorities toward rumors for the evidence retrieval subtask.
The remainder of the paper is organized as follows. We review the literature in Section 2 and formally define our task in Section 3. In Section 4, we discuss our dataset construction approach. Our experimental design and setup are presented in Sections 5 and 6, respectively. We analyze our experimental results and answer our research questions in Section 7. We conclude and suggest some future directions in Section 8. Finally, we discuss the limitations of our work in Section 9.
2https://github.com/Fatima-Haouari/AuRED 3Our released resources are presented in Appendix B.4.

heading: 2 Related Work
text: In this section, we review previous studies on rumor verification in social media and fact-checking.
Rumor Verification in Social media. There exists a considerable body of literature on rumor verification in social media (Ma et al., 2018; Kumar and Carley, 2019; Yu et al., 2020; Choi et al., 2021; Bai et al., 2022a). The majority of prior research has leveraged the propagation networks such as the structure of replies (Ma et al., 2018; Bian et al., 2020; Haouari et al., 2021; Bai et al., 2022b), stance of replies (Zubiaga et al., 2016; Derczynski et al., 2017), or retweeters metadata (Liu and Wu, 2018). In addition to the propagation networks, incorporating evidence from the Web was proposed by Dougrez-Lewis et al. (2022) and Hu et al. (2023). Moreover, Haouari and Elsayed (2023, 2024) proposed recently leveraging the stance of authority tweets towards rumors.
Although there are many studies, the research in Arabic Rumor verification remains limited. Previous studies have almost exclusively utilized the tweet textual content for verification (Elhadad et al., 2021; Mahlous and Al-Laith, 2021; Al-Yahya et al., 2021; Alqurashi et al., 2021; Sawan et al., 2021). Recently, Haouari et al. (2021) leveraged the replies structure, Althabiti et al. (2022) incorporated the detected sarcasm and hate speech in the replies, while Albalawi et al. (2023) exploited the images and videos embedded in the rumor tweet. Differently, in our work we propose using the evidence tweets retrieved from the authority timelines.
Most of the existing datasets (refer to Appendix A) focus on using the propagation networks as evidence, while the majority of Arabic Rumor verification datasets do not incorporate any external evidence. Compared to existing datasets, AuRED incorporates evidence from authority timelines.
Fact-Checking. Claim verification using evidence from Wikipedia was introduced as part of
FEVER shared task by Thorne et al. (2018a). The task is a pipeline of three subtasks namely documents retrieval, evidence selection, and claim verification. A plethora of studies addressed the task contributing either to the evidence retrieval or claim verification or both. For evidence retrieval, existing studies either adopted neural ranking models (Hanselowski et al., 2018; Zhou et al., 2019; Nie et al., 2019a,b) or pre-trained models (Liu et al., 2020; Jiang et al., 2021; DeHaven and Scott, 2023). Recent studies, exploited pre-trained models but with variant loss functions and some additional enhancements. Some addressed the task as a binary classification task (Zhong et al., 2020; Si et al., 2021; Jiang et al., 2021; DeHaven and Scott, 2023), some proposed a pairwise ranking model (Soleimani et al., 2020; Liu et al., 2020), while others explored distance-based loss functions (Bekoulis et al., 2021). For the claim verification task, most of the studies formulated it as a graph-based reasoning task (Zhou et al., 2019; Liu et al., 2020; Zhong et al., 2020; Park et al., 2022). Others, proposed incorporating the topic and implicit stance of evidence using the capsule network (Si et al., 2021; Ma et al., 2022), or multilevel attention (Kruengkrai et al., 2021).
In our work, we consider evidence retrieval from authority timelines and rumor verification using evidence from authorities similar to the evidence selection and the claim verification for factchecking tasks respectively. Moreover, we investigate the knowledge transfer ability of existing factchecking datasets to our task. One of such datasets is FEVER (Thorne et al., 2018a), an English fact checking dataset containing 185,445 claims, and their relevant evidence sentences from Wikipedia.

heading: 3 Task definition
text: We propose the task of Rumor Verification using Evidence from Authorities with two subtasks:
• Evidence Retrieval: Given a rumor expressed in a tweet and a set of authorities for that rumor, the system should retrieve evidence tweets posted by any of those authorities. An evidence tweet is a tweet that can be further used to detect the veracity of the rumor. The set of authorities has one or more authority Twitter accounts, represented by a list of tweets from their timelines that are posted during the period surrounding the rumor.
• Rumor Verification: Based solely on the evidence tweets retrieved by the above subtask, determine if the rumor is supported (true), refuted (false), or unverifiable (in case not enough evidence to verify it exists).

heading: 4 AuRED Dataset
text: To expedite the development of automatic verification systems and to evaluate proposed models for our task, we introduce the first Authority-RumorEvidence Dataset (AuRED). We target Arabic as it is one of the most used languages in Twitter (Alshaabi et al., 2021), yet under-explored for rumor verification. As presented in Figure 2, the dataset was constructed by annotating a set of rumors, selected from two existing datasets (Section 4.1) following two main steps (1) finding authorities that can help verify the rumors (Section 4.2), and (2) collecting and annotating the timelines of those authorities to find evidence tweets (Section 4.3).

heading: 4.1 Rumors Collection
text: Due to time and budget constraints, we randomly selected 160 rumors from AuFIN (Haouari et al., 2023) and AuSTR (Haouari and Elsayed, 2024) datasets. AuFIN is an Arabic test collection for authority finding in Twitter, where each rumor is associated with its relevant authorities. AuSTR is an Arabic dataset for detecting the stance of authorities towards rumors. Given that all AuFIN rumors were collected originally from a fact-checking Website, it lacks true (i.e., confirmed) rumors as factcheckers focus mainly on verifying false (i.e., denied) rumors, we had to get all of our 30 true rumors from AuSTR dataset. Moreover, we selected 31 false rumors from AuSTR, as each has already at least one authority tweet refuting it. In total, 99 (61.9%) of our rumors are from AuFIN while 61 (38.1%) are from AuSTR.

heading: 4.2 Authority Finding
text: Finding authorities in Twitter for a specific rumor was proposed recently by Haouari et al. (2023). They define an authority for a specific rumor as an entity having the real knowledge or power to verify or deny that rumor. For example, if the rumor is about a health issue in Iraq, then the health minister, ministry, or other leaders in health organizations in Iraq are potential authorities.
AuFIN rumors are already associated with their relevant authorities, however AuSTR rumors are only associated with an authority tweet either supporting, refuting or irrelevant to the rumor. Therefore, for AuSTR rumors, in addition to considering the authority of the associated authority tweet, we collected more authorities for each rumor following the same approach proposed by Haouari et al. (2023). Two annotators, a PhD holder and a PhD candidate, performed the task independently, then met to discuss their annotations. Only potential authorities that both annotators agreed upon during their meeting were kept in AuRED.

heading: 4.3 Evidence Annotation
text: In the context of this work, we consider the rumor tweet as a pointer to the period of the rumor propagation, assuming that the rumor is circulating for a few days before and/or after the time at which the tweet containing it is posted. Therefore, for evidence annotation, we limit the authority timelines to the tweets within 3 days before and after the posting time of the rumor tweet. The timelines were collected using the Academic Twitter search API which facilitates collecting user historical timelines.4 We carried out two stages for evidence extraction: (a) Annotation: Following our annotation guidelines, one annotator labeled all tweets in all authority timelines as supporting, refuting, or carrying not enough info towards the corresponding rumor tweet (constituting AuRED core dataset). To measure the quality of our data, and to have a double-annotated sample, a second annotator then labeled solely one authority timeline per rumor (constituting AuRED* subset). To ensure the inter-annotator consistency, we asked the annotators to ask themselves this general question: If I was given the authority tweet, do I have a strong evidence to decide if the rumor is true (supported),
4https://developer.x.com/en/docs/ twitter-api/tweets/search/api-reference/ get-tweets-search-all
false (refuted), or unverifiable (Not enough information to verify it). At the end of this stage, we measured the data quality of AuRED* using Cohen’s Kappa for inter-annotator agreement (Cohen, 1960) as 0.67, which indicates “substantial” agreement (Landis and Koch, 1977). It is worth noting, that any disagreement between the annotators was then resolved in the next stage. (b) Resolving Disagreements: As a final step, both annotators met to discuss and resolve any disagreements in AuRED*, and hence decide the final labels. The statistics about our AuRED and AuRED* are presented in Table 1. We present examples from AuRED, our annotation challenges, and some data analysis in Appendix B.

heading: 5 Experimental Design
text: Our task is closely related to the general task of fact-checking (Thorne et al., 2018b). In fact, it can be viewed as a special case of the fact-checking task, where evidence for verification is exclusively retrieved from authorities rather than from any other source, e.g., Web pages, or posts from layman users or propagation networks on social media. With a large body of existing research on the fact-checking task (Nakov et al., 2021), it is intriguing to investigate how existing evidence retrieval and claim verification models, originally designed for the general fact-checking task, perform on our specific task. Moreover, with the availability of datasets for the general task in other languages
(e.g., FEVER (Thorne et al., 2018a)), it is then intuitive to explore the potential of cross-lingual transfer learning. Accordingly, we address the following research questions:
• RQ1: How effective are the existing models for our task under the cross-lingual zero-shot setup? • RQ2: How do existing models perform on our task if they are directly fine-tuned with AuRED?
It is worth noting that for each of the two research questions, we evaluate the performance of the models on AuRED for the two sub-tasks. Accordingly, to address both questions, we design our experiments as follows:
• Cross-lingual Zero-shot Setup: We study the performance of existing models on AuRED when they are fine-tuned only on English data for evidence retrieval and rumor verification, without being fine-tuned on AuRED. • In-domain Fine-tuning Setup: We study the performance of existing models on AuRED when they are directly fine-tuned on AuRED.

heading: 6 Experimental Setup
text: In this section, we present our detailed experimental setup. We discuss our adopted evidence retrieval and rumor verification models in Sections 6.1 and 6.2, respectively. We also discuss how we evaluate those models in Section 6.3.

heading: 6.1 Evidence Retrieval Models
text: In addition to evaluating strong sparse and dense retrieval approaches, we selected two SOTA models (KGAT and MLA) for evidence retrieval which exhibited the best performance on FEVER test set (Park et al., 2022).5 Moreover, we explore a model with a distance-based loss function. Finally, we adopted a stance-based approach for evidence retrieval. It is worth noting that although 49.05% of AuRED evidence tweets are multimodal, all the models we adopted in this work considers only the textual content of the tweets. In this section, we present the models and their implementation details.
1. BM25: One of the most successful lexical retrieval models (Jones et al., 2000). Using
5The model proposed recently by DeHaven and Scott (2023) is SOTA but they adopt re-retrieval using hyperlinks in retrieved sentences to beat KGAT. Re-retrieval is not applicable in our work
Pyserini (Lin et al., 2021), we constructed an index for all tweets from all authorities for a each rumor. We then retrieved, for each rumor, the top relevant authority tweets from the corresponding index.
2. mContriever (Izacard et al., 2021): A multilingual dense retrieval model that achieves good retrieval performance on Arabic data when further fine-tuned using MS MARCO dataset. For each rumor, we retrieved tweets that are the closest in the Contriever’s embedding space using cosine similarity.
3. KGAT (Liu et al., 2020): A widely adopted retrieval model in fact-checking studies (Zhao et al., 2019; Park et al., 2022; Ma et al., 2022; Chen et al., 2022). It is a pairwise BERTbased model where the margin ranking loss is adopted to maximize the distance between the positive and the negative claim-sentence pairs. As suggested by the authors, the model during training was fine-tuned to maximize the distance between each positive and negative rumor-tweet-authority-tweet pairs for all authority tweets for a specific rumor. At inference, the score predicted for each rumortweet-authority-tweet pair is used to retrieve the top evidence tweets. We adopted the authors’ implementation.6
4. MLA (Kruengkrai et al., 2021): A pointwise BERT-based binary classifier to detect evidence vs. non-evidence. The cross entropy loss was adopted. For negative examples, the authors proposed sampling M non-evidence sentences from the labelled documents and M from retrieved potentially-relevant documents, where M is twice the number of evidences. In our work, we only have the labelled documents (timelines), so we considered the number of non-evidence tweets to be 4 times the number of evidence tweets for each rumor.7 At training and inference, rumor-tweetauthority-tweet pair are fed to a BERT-based model separated by a [SEP] token. The authors’ code was adopted for our experiments.8
5. TML (Bekoulis et al., 2021): We investigate the performance when adopting the triplet
6https://github.com/thunlp/KernelGAT 7Based on our preliminary experiments we found that 4 is
the best considering 2, 4, 6, and 8 when fine-tuning 8https://github.com/nii-yamagishilab/mla
margin loss (TML), compared to the pointwise (MLA) and the pairwise (KGAT) models. This loss minimizes the pairwise distance between the rumor and the evidence, and maximizes the distance between the rumor and non-evidence. As suggested by the authors, the evidence and the non-evidence tweets are prepended with the rumor and a [SEP] token. During inference, the pairwise distance is computed between each rumor and its corresponding authority tweets (prepended by rumor [SEP]) to select the top with the lowest distance. We adopted the authors’ code.9
6. STAuRED: Motivated by the task of detecting the stance of authorities (Haouari and Elsayed, 2023, 2024) as a source of evidence, we finetuned BERT-based stance detection model using AuRED to classify whether an authority tweet SUPPORTS, REFUTES, or NOT ENOUGH INFO. We feed BERT the rumor tweet as sentence A and the authority tweet as sentence B separated by the [SEP] token. Finally, we use the representation of the [CLS] token as input to a single classification layer with three output nodes, added on top of BERT architecture, to compute the probability for each stance class. For retrieving the top evidence tweets, we considered the sum of the softmax scores of both SUPPORTS and REFUTES labels as a reranking score.
Implementation details: For evaluation, we adopted a cross validation setup where we split our AuRED dataset into 5 folds, each containing 32 rumors ensuring balance across rumors labels. We fine-tuned the models using 3 folds and we selected the best model based on Mean Average Precision (MAP) on the dev set for each fold. We fine-tuned using 4 different learning rates [2e-5, 3e-5, 4e-5, 5e-5]. We trained all the models for 5 epochs using a batch of size 8. As our dataset contains tweets only, we adopted MARBERTv2 (Abdul-Mageed et al., 2021),10 an Arabic BERT model pre-trained using 1 billion Arabic tweets. For the cross-lingual evidence retrieval setup, we adopted the original setup suggested by the authors, i.e., fine tuning the models with English FEVER (Thorne et al., 2018a), but we replaced the English BERT with multilin-
9https://github.com/bekou/evidence_aware_ nlp4if
10https://huggingface.co/UBC-NLP/MARBERTv2
gual BERT (mBERT) (Devlin et al., 2019).11 We retrieved the top 5 evidence tweets for each rumor.

heading: 6.2 Rumor Verification Models
text: To have a full pipeline for both evidence retrieval and rumor verification, in our experiments we adopted both MLA and KGAT where models for both subtasks were proposed by the authors:12
1. MLA (Kruengkrai et al., 2021): It adopts multi-task learning considering the verification as the main task, and evidence retrieval as an auxiliary task where it incorporates the evidence retrieval scores through joint training. The model applies token-level attention over a claim-evidence pair, token and sentence-level self-attentions for evidence sentences. Finally, it combines all hidden states with the evidence retrieval scores at the final attention layer.
2. KGAT (Liu et al., 2020): A Kernel Graph Attention Network that utilizes the retrieved evidence to construct a fully connected graph and perform reasoning to verify the claims. Each node in the graph is represented using the [CLS] token of a pre-trained BERT, by feeding it a concatenation of the claim and the evidence separated by a [SEP] token.
Implementation details: During training, Both MLA and KGAT prepend the gold evidence (decided by the annotators) to the retrieved evidence, and take as input both the rumor and 5 evidence tweets. At inference time, only the retrieved evidence is considered to verify the rumors. We adopted the same cross validation setup adopted for evidence retrieval, but we fine-tuned the models based on the best Macro-F1 on the dev set.

heading: 6.3 Evaluation Scenarios and Measures

heading: 6.3.1 Evidence Retrieval
text: To evaluate the performance of the evidence retrieval models, we considered two sets of measures based on two scenarios as presented below:
• The User Scenario is the case where a human, mostly a fact-checker, is directly interacting with the evidence retrieval component
11https://huggingface.co/ bert-base-multilingual-uncased
12The model proposed recently by DeHaven and Scott (2023) is SOTA for claim verification for FEVER, but they adopted DeBERTa V2 XL MNLI, which is not available for Arabic. Moreover, we could not adopt their retrieval model due to the re-retrieval step which is not applicable to our task.
Table 2: Performance of Cross-lingual Zero-shot Evidence Retrieval. Bold scores are the best for each
test set. Standard and FEVER scores to evaluate the user and system scenario respectively.
test set.
Test Set Verification model m-F1 Strict m-F1
AuRED MLA 0.215 0.171KGAT 0.422 0.413
AuRED* MLA 0.226 0.196KGAT 0.426 0.417
to get evidence that can help her verify a given rumor. In such scenario the system should retrieve as much evidence, preferably from different authorities, as possible to convince the user. Therefore, the system is required to provide a ranked list of potentially-evidence tweets. To measure the ability of the system to retrieve evidence tweets higher in the list, we adopt the standard information retrieval rank-based measure Mean Average Precision (MAP), and we report Recall@5 (R@5). • The System Scenario is the case where the output of the retrieval component is used automatically by the down-stream rumor verification component. In this scenario, retrieving at least one evidence tweet for the given rumor might be enough. Hence we consider the evaluation measures adopted by the FEVER shared task (Thorne et al., 2018b), namely Macro R@5, where an instance is scored if at least one evidence is retrieved, and we report Macro P@5, and F1@5 computed using both these metrics.

heading: 6.3.2 Rumor Verification
text: To evaluate the performance of rumor verification models, we adopt Macro-F1 measure to account for the label imbalance in our data. Inspired by FEVER score which adopts strict label accuracy (Thorne et al., 2018b), we also adopt strict Macro-F1, where we consider the label correct only if at least one
correct evidence is retrieved by the adopted evidence retrieval model. Specifically, we consider an instance a false positive if the label is predicted correctly but no single correct evidence was retrieved.

heading: 7 Results and Discussion
text: In this section, we present and discuss the results of our experiments which address the two research questions introduced in Section 5.

heading: 7.1 Cross-lingual Zero-shot Scenario (RQ1)
text: For this setup, we fine-tuned MLA and KGAT models presented in Section 6.1 and Section 6.2 using the authors’ setup for both evidence retrieval and claim verification tasks. Since, for this scenario, we train on English data (FEVER) and test on Arabic data (AuRED), we adopted mBERT as the pretrained model. The models were then used to retrieve evidence for AuRED test rumors and verify them using the retrieved evidence. We report the average performance, using cross-validation, for evidence retrieval and rumor verification in Table 2 and Table 3 respectively.
Evidence Retrieval: As shown in Table 2, MLA achieved better performance than KGAT for evidence retrieval across all evaluation measures on both AuRED and AuRED*. Given that this setup is both cross-lingual (training and testing on two different languages -English vs. Arabic-) and crossdomain (training and testing on two different domains -Web pages vs. tweets-), we believe the performance is acceptable. It also indicates the potential of knowledge transfer using FEVER dataset to our task for evidence retrieval. Looking at the recall performance, we also note that MLA was able to retrieve about an average of 59% of the evidence tweets over all rumors, and at least one evidence tweet for about 76% of them. The latter in particular is important for the system scenario, where the evidence is used in the verification downstream task. Overall, the models performed better on AuRED* than AuRED in terms of MAP and recall. This is somewhat expected as AuRED* is less challenging because evidence is retrieved from the timeline of a single authority for each rumor.
Rumor Verification: As presented in Table 3, the performance of both models is considered poor, which we speculate due to the domain difference. We believe the way authorities refute or support rumors in their tweets differs significantly in terms of writing from how Wikipedia sentences refute or
support claims (refer to Table 7 in Appendix B.1). Recall that FEVER claims are generated by manipulating the Wikipedia sentences adopting paraphrasing, negation, or entity substitution to name a few changes (Thorne et al., 2018a). Thus, the models may have learned different styles of evidence to decide whether a given a rumor REFUTES, SUPPORTS, or NOT ENOUGH INFO to verify it. Finally, we observe that KGAT significantly outperforms MLA in verification, despite the superiority of the latter in evidence retrieval, showing clearly that the retrieval and verification models are different.

heading: 7.2 In-domain Fine-tuning Scenario (RQ2)
text: For this setup, we tested the evidence retrieval and rumor verification models presented in Section 6.1 and Section 6.2 respectively. We fine-tuned all the models using AuRED. The performance on evidence retrieval and rumor verification is presented in Table 4 and Table 5 respectively.
Evidence Retrieval: MLA and STAuRED are the best performing models in terms of the standard MAP and R@5 measures on both AuRED and AuRED*. The performance of STAuRED in particular highlights the potential of detecting the stance for evidence retrieval. However, surprisingly, BM25 (the lexical retrieval model) is the best performing model in retrieving evidence for more rumors, as indicated by the FEVER scores, on both AuRED and AuRED*. Recall that FEVER measures reward models that cover more rumors (by retrieving at least one evidence) higher than models that retrieve more evidence. This result indicates that lexical retrieval is probably enough to provide minimum evidence, however that might not be sufficient for fact-checkers who are interested in more evidence to reach a solid verification decision.
Rumor Verification: Neither of the models perform well on this task, indicating a huge room for improvement. One of the main reasons is the small number of training rumors in AuRED; in fact, only 96 rumors (constituting 3 folds) were used for training. There are multiple solutions to address this problem in the future including data augmentation, e.g., using synthetic data that is automatically generated by large language models (Ubani et al., 2023) or seq2seq text generation models (Pan et al., 2023), or domain adaptation (Yue et al., 2023) over fact checking datasets. While KGAT still exhibits better performance than MLA when fine-tuned with the in-domain training data, the performance interestingly has not reached the performance un-
Table 4: Performance of In-domain Fine-tuning for Evidence Retrieval. Bold and underlined scores are the
best and second-best respectively for each test set. Standard and FEVER scores to evaluate the user and system scenario respectively.
test set.
Test Set Verification model m-F1 Strict m-F1
AuRED MLA 0.351 0.324KGAT 0.371 0.342 AuRED* MLA 0.354 0.339KGAT 0.366 0.348
der the cross-lingual setup shown in Table 3. This can be attributed to the size of the training data in both cases; the big collection of claims in FEVER (145,449 training claims) enabled KGAT to better learn reasoning for the verification task. We will leave the investigation of such result to future work.

heading: 8 Conclusion and Future Work
text: In this paper, we introduced the new task of rumor verification using evidence from authorities over Twitter. We constructed and released the first Authority-Rumor-Evidence Dataset (AuRED) which consists of 160 rumors expressed in tweets and 692 timelines of authorities Twitter accounts comprising about 34k annotated tweets in total. We explore existing fact-checking models to set up the baseline systems for our two substasks namely evidence retrieval and rumor verification. Our experiments show that evidence retrieval models for fact-checking achieved competitive benchmark results even under cross-lingual zero-shot setup, however the performance on rumor verification is still far from enough. For future work, we plan to (1) consider the multimodality of evidence tweets to
improve the evidence retrieval, (2) augment the dataset to expand the number of rumors to improve the rumor veracity prediction, (3) propose models to improve the performance achieved on both subtasks, and (4) construct a similar dataset in English to facilitate and encourage research on the task.

heading: 9 Limitations
text: Due to time and budget constraints, this work is limited in two aspects as presented below:
Data Size. The small number of rumors in our data, despite being traditionally reasonable for retrieval tasks, make it very challenging for the rumor verification task in particular. This motivates the need to build models with the ability to transfer knowledge from relevant datasets. However, because we are targeting the Arabic language this raised another limitation due to the limited Arabic resources for fact checking and evidence-based rumor verification. Moreover, we believe data augmentation with real or synthetic data can improve the performance of the models.
Evidence multimodality. Although, evidence is not textual in 38.5% of the evidence tweets, we did not consider the multimodality in this work. Considering multimodal evidence retrieval models (Hu et al., 2023; Yao et al., 2023) or expanding the context of the rumor with extracted text from images, videos, or external news articles embedded in the authority tweets can further improve the retrieval of evidence tweets.

heading: Acknowledgments
text: The work of Fatima Haouari was supported by GSRA grant #GSRA6-1-0611-19074 from the Qatar National Research Fund (a member of Qatar Foundation). The work of Tamer Elsayed was made possible by NPRP grant #NPRP-11S-1204170060 from the Qatar National Research Fund. The statements made herein are solely the responsibility of the authors.

heading: A Comparison with Other Datasets
text: As presented in Table. 6, we review existing rumor verification datasets in terms of the evidence adopted for rumor verification. As shown in the table, most of existing studies focus on using the propagation networks. Some of the studies relied on the rumor textual content solely without any external evidence for verification (Alsudias and Rayson, 2020; Albalawi et al., 2023; Ameur and Aliane, 2021; Elhadad et al., 2021). Recently, some studies incorporated evidence from the Web such as relevant Web articles (Hasanain et al., 2020;
Dougrez-Lewis et al., 2022; Hu et al., 2023) and images (Hu et al., 2023) in addition to social media users’ metadata (Nielsen and McConville, 2022). Most of the existing datasets for Arabic Rumor verification do not incorporate any external evidence. Some notable exceptions are the data released by Haouari et al. (2021) and Hasanain et al. (2020) who incorporated the propagation networks and Web articles as external evidence respectively. Compared to existing datasets, AuRED incorporates evidence from authority timelines.

heading: B Data Overview
text: In this section, we present some examples from our dataset (B.1), discuss our data annotation challenges (B.2), show some analysis about our dataset (B.3), and finally we present the resources we release (B.4).
B.1 Data Examples We present example rumors and corresponding evidence tweets from our AuRED dataset in Table 7.
B.2 Annotation Challenges There are several challenges associated with annotating the data. We elaborate on a few of them through discussing the rumor tweet “Urgently, giving the Corona vaccine has stopped urgently in the Kingdom of Saudi Arabia. There is no power or strength from God. Five people died after receiving the vaccine.” Multiple rumors: A tweet may contain multiple potential rumors. For example, our tweet contains two potential rumors as a result of receiving the new Corona virus vaccine: (a) “vaccine has stopped urgently in the Kingdom of Saudi”, and (b) “Five
people died after receiving the vaccine”. We asked annotators to focus on the rumor that had been already fact-checked by our sources (e.g., rumor (b) is verified by “Misbar” fact checking platform, assuming those are viral, consequently could have higher impact on the community.
(a) Image Evidence
Relevant people
، "ىمسرلا ا *ٮ,ٮاسح "ى*ڡ ك,ٮ ًلاهأ ،ا *ٮ7ٮيولوأ ك7ٮحص :لض *ڡ7ٮ ةAٮ ,ٮط ةراش7ٮسا وأ راس *ڡ7ٮسا كAٮدل تامولعملا كمه7ٮ |
:لض *ڡ7ٮ ،ةAٮحصلا
ة"ٮدوعسلا ةحصلا ةرازو $SaudiMOH Follow
$SaudiMOH937 $LiveWellMOH
"ىملاعلإا لمعلل "ىموكحلا لصاو7ٮلا زكرم كر7ٮشملا
0ىموكحلا لصاو7ٮلا $CGCSaudi
Follow
Trends for you يلودلا دمح راطم Trending in Qatar
دحاو_بعش_دحاو_شیج# 2,069 posts
Trending in Qatar
Imran Khan 304K posts
Politics · Trending
هیطغتلا_جراخ_نادوسلا# 14.5K posts
Business & finance · Trending
ندرلاا_رطق# 19.2K posts
Trending in Qatar
PMLN 8,772 posts
Politics · Trending
يرطقلا روھمجلا Trending in Qatar
رصنلا_للاھلا# 217K posts
Sports · Trending
Show more Terms of Service Privacy Policy Cookie Policy Accessibility Ads info © 2024 X Corp.More
Search
ة"ٮدوعسلا ةحصلا ةرازو reposted 0ىموكحلا لصاو7ٮلا $CGCSaudi
Translated from Arabic by
؟ا#ٮوروك سور+ٮ #ڡ حا /ڡل ذ #حأ نم فو #ح/ٮ+ٮ نمل ةعي>ٮرلا ق+ٮ #ڡو/ٮ رو/ٮكدلا ةحصلا_ريزو# ه >حو+ٮ اذام يموكحلا_لصاوتلا# $SaudiMOH What does ةحصلا_ريزو# Dr. Tawfiq Al-Rabiah advise those who are afraid of taking the Corona virus vaccine? يموكحلا_لصاوتلا#$SaudiMOH
0:18 / 0:59
1:00 PM · Dec 18, 2020
96 656 1K 33
Post your reply Reply
More Tweets
38 115 1K 611K
Post
Home Explore Notifications Messages Lists Bookmarks Communities Premium Profile More
Post
مير @rasuwaifeh Messages
(b) Video Evidence
Relevant people
، "ىمسرلا ا *ٮ,ٮاسح "ى*ڡ ك,ٮ ًلاهأ ،ا *ٮ7ٮيولوأ ك7ٮحص :لض *ڡ7ٮ ةAٮ ,ٮط ةراش7ٮسا وأ راس *ڡ7ٮسا كAٮدل تامولعملا كمه7ٮ |
:لض *ڡ7ٮ ،ةAٮحصلا
ة"ٮدوعسلا ةحصلا ةرازو $SaudiMOH Follow
$SaudiMOH937 $LiveWellMOH
Trends for you يلودلا دمح راطم Trending in Qatar
دحاو_بعش_دحاو_شیج# 2,069 posts
Trending in Qatar
Imran Khan 304K posts
Politics · Trending
هیطغتلا_جراخ_نادوسلا# 14.5K posts
Business & finance · Trending
ندرلاا_رطق# 19.2K posts
Trending in Qatar
PMLN 8,805 posts
Politics · Trending
ضايرلا 193K posts
Sports · Trending
يرطقلا روھمجلا Trending in Qatar
سب_لب# 1,019 posts
Trending in Qatar
Search
ة"ٮدوعسلا ةحصلا ةرازو $SaudiMOH
Translated from Arabic by
:انوروك_حاقل# نمآ $ى &ٮا )حم
عيم )حلل ر &ڡو3ٮم .حا 3ڡللا ذ &حلأ يتحص_قیبطت# ر )ٮع نلآا ل )حس
انوروك_حاقل# : Safe Free Available to everyone Register now via يتحص_قیبطت# to take the vaccine.
تيحص | Sehhaty تيحص | Sehhaty Health & Fitness
Install

heading: 2:36 PM · Dec 19, 2020
text: 0ىع"ٮ 4ٮسلا م"ٮ 6ٮر و4ٮأ ·@sfr_55 Dec 19, 2020 ده ,حلا ولذ,ٮ ورص7ڡام اهمكاحو ا *ٮدل,ٮ "ىمحAٮ اللهو ءاAٮ *ٮدلا "ى*ڡ شي يا نم لض *ڡا وه الله
حا 7ڡلاه نود,ٮ ا *ٮAٮمحAٮ اللهو
حلاص ن4ٮ اللهد4ٮع ن4ٮ نام"ٮلس د ·@a]sa]eh_s2^2^ Dec 19, 2020 ..مكAٮلع ملاسلا
.ت,ٮسلا موAٮلا دعوملا "ى *ٮا ,ح و ىلولاا ةعر ,حلل دعوم ذ *حلأ ةعم ,حلا سما تل ,حس نم ه*ٮا دع,ٮ امAٮ *ڡ تملع و نAٮعو ,ٮسا ل,ٮ 7ڡ ةAٮمسوملا از*ٮول *ڡ *ٮلاا دض مAٮعط7ٮ تذ *حا ق,ٮس ا *ٮا
..عي,ٮاسا ةع,ٮرا ةدم نAٮمAٮعط7ٮلا ني,ٮ نوكAٮ نا لض *ڡلاا .موAٮلا رضحا نل هAٮلع ءًا *ٮ,ٮ و نكم7ٮا مل نكل ٩٣٧ قيرط نع دعوملا ءا qعلا تلواح
1 2 1
Al-Mujadilah | ةلدا 4حملا @a]mujadi]ah_qa Ad We provide opportunities for women to amplify their spiritual, intellectual, and emotional capacity. Explore what Al-Mujadilah offers you.
1 10 60 280K
Dr/Sabry(M.D.vascular surgery) ·$Msabrydr Dec 20, 2020 نم سي*ڡ*ٮ ا *ٮاو تي ,حوز كلذكو ل,ٮ 7ڡAٮلا تي*ٮ,ٮ ل *حدا "ى ,حا "ى *ٮلا نAٮع,ٮا7ٮلا ه *ڡاضا ل,ٮ 7ڡAٮلا قAٮ ,ٮط7ٮلا ذ *حلا بلط تمد7ڡو قAٮ ,ٮط7ٮ ا *ٮآ ع,ٮ7ٮ "ىحصلا عاط 7ڡلا,ٮ نAٮلماعلا نم "ى *ٮلا تل ,حس ماAٮا سم *ح
ام كل7ٮ روملاا حضويو بوا ,حAٮ لوئسم دح نكمم ءار ,حلاا تح7ٮ بلطلا لازام نكلو مAٮعط7ٮلا لحلا
Show more replies
More Tweets
ة"ٮدوعسلا ةحصلا ةرازو ·$SaudiMOH Feb 6 .2024 ماع "ى*ڡ "ىحصلا عاط 7ڡلا تاAٮلاع *ڡو ثادحأ زر,ٮأ فش7ٮكا
Post
Home Explore Notifications Messages Lists Bookmarks Communities Premium Profile More
Post
(c) Implicit Evidence
Time sensitive rumors: The factuality of some rumors may change within a short period of time. For example, the COVID tolls (e.g., deaths) in our example could increase or decrease over time if the rumor is true, hence, we urged the annotators to consider the tweet timestamp while annotating. Context of evidences: Verifying rumors requires looking at the authority timelines entirely rather than reading tweets independently. For instance, verifying the number of COVID tolls could require summing up the number of cases in an authority timeline within a time window. Multimodality of evidences: Evidence could be extracted from text, images, videos, or a combination of these. The Saudi Ministry of Health posted several tweets that are useful for verification but not all of them contain textual evidences. Figure 3a shows an image of the highlights of the press conference of spokesman of the Saudi Ministry of Health. The spokesman announced the beginning of the vaccine campaign and encouraged people to register to take the vaccine which denies both rumor (a) and (b). On the other hand, Figure 3b shows a video of the health minister confirming the safety of the vaccine and denying the rumors about its side effects. The tweet also contains an implicit textual evidence that calms the public down. Accordingly, we asked annotators to carefully analyze the media not only the tweet text which required extra time and effort. Implicit evidences: The evidences in authority timelines are not always stated explicitly. For example, Figure 3c shows a tweet from the Saudi Ministry of Health encouraging people to book vaccination appointments. Without an explicit statement, this tweet denies both rumor (a) and (b). We highly urged the annotators to consider all potential evidences including implicit ones.
B.3 Data Analysis
To show the quality of AuRED, we analyzed its coverage and diversity to ensure the generalizability of models trained on it. In the following we discuss different aspects. Dialectical/Geographical Coverage: AuRED contains rumors that are of interest to different Arab countries such as Egypt, Qatar, Saudi Arabia, Kuwait, among other countries. Figure 4 shows the geographical distribution of rumors across the Arab countries. The dataset also covers rumors of interest to the Arab users although not happening in the Arab region. Such geographical coverage implies the coverage of diverse dialects in AuRED. We used ASAD tool (Hassan et al., 2021) to automatically analyze the dialectical coverage of the tweets in AuRED. We found 92.5% of tweets are written in Modern Standard Arabic (MSA) and the remaining are dialectical tweets.
Domain Coverage: We define domain here as the topic of the rumor such as politics, health, sports,
etc. Figure 5 shows the diverse coverage of domains of rumors in AuRED. Multimodality: To support the development of versatile verification systems, AuRED is labeled for different types of evidences, i.e., text, and or media. It contains 49.05% multimodal evidence tweets, 38.5% of which are media evidences that show the insufficiency of text for rumor verification. The remaining contain both text and media that complement each other for rumor verification.
B.4 Data Release
We release the following data as part of AuRED, taking into consideration the content distribution policy:13
• Rumors: 160 rumors expressed in tweets each labeled as SUPPORTES, REFUTES, or NOT ENOUGH INFO. We release the rumor IDs and tweets text.
• Authorities timelines: Each rumor is associated with timelines of potential authorities. We release the authority Twitter account link, tweets IDs, and tweets text.
• Evidence tweets: Each rumor is associated with the evidence tweets IDs and text.
• Authorities tweets media: The images and videos extracted from authority tweets.
• Data folds: To enable consistent benchmarking on the dataset, we provide our data folds. i.e., 5 folds we adopted for our crossvalidation setup.
13https://developer.twitter.com/en/ developer-terms/agreement-and-policy

heading: Future Work
text: For future work, we plan to (1) consider the multimodality of evidence tweets to
improve the evidence retrieval, (2) augment the dataset to expand the number of rumors to improve the rumor veracity prediction, (3) propose models to improve the performance achieved on both subtasks, and (4) construct a similar dataset in English to facilitate and encourage research on the task