SlowGuess/NAACL_2022_OCR huggingface dataset:
The focus of this work, as well as in many previous studies (Ma et al., 2017, 2018; Khoo et al., 2020; Bian et al., 2020), is rumors on social media, not just the "fake news" strictly defined as a news article published by a news outlet that is verifiably false (Shu et al., 2017; Zubiaga et al., 2018). To our knowledge, there is no public dataset available for classifying propagation trees in tweets about COVID-19, where we need the tree roots together with the corresponding propagation structure, to be appropriately annotated with ground truth. In this paper, we organize and construct two datasets Weibo-COVID19 and Twitter-COVID19 for experiments. For Twitter-COVID19, the original dataset (Kar et al., 2020) of tweets was released with just the source tweet without its propagation thread. So we collected all the propagation threads using the Twitter academic API with the twarc2 package in python. Finally, we annotated the source tweets by referring to the labels of the events they are from the raw COVID-19 rumor dataset (Kar et al., 2020), where rumors contain fact or misinformation to be verified while non-rumors do not. For Weibo-COVID19, data annotation similar to Ma et al. (2016), a set of humorous claims is gathered from the Sina community management center and non-rumorous claims by randomly filtering out the posts that are not reported as rumors. Weibo API is utilized to collect all the repost-reply messages towards each claim. Both Weibo-COVID19 and Twitter-COVID19 contain two binary labels: Rumor and Non-rumor. For Weibo-COVID19 as the target dataset, we use the TWITTER dataset (Ma et al., 2017) as the source data in our low-resource (i.e., cross-domain and cross-lingual) settings; In terms of Twitter-COVID19 as the target dataset, we use WEIBO (Ma et al., 2016) as the source data. The statistics of the four datasets are shown in Table 3.
We set the number  $L$  of the graph convolutional layer as 2, the trade-off parameter  $\alpha$  as 0.5, and the adversarial perturbation norm  $\epsilon$  as 1.5. The temperature  $\tau$  is set to 0.1. Parameters are updated through back-propagation (Collobert et al., 2011) with the Adam optimizer (Loshchilov and Hutter,
2018). The learning rate is initialized as 0.0001, and the dropout rate is 0.2. Early stopping (Yao et al., 2007) is applied to avoid overfitting. We run all of our experiments on one single NVIDIA Tesla T4 GPU. We set the total batch size to 64, where the batch size of source samples is set to 32, the same as target samples. The hidden and output dimensions of each node in the structure-based network are set to 512 and 128, respectively. Since the focus in this paper is primarily on better leveraging the contrastive learning for domain and language adaptation on top of event-level representations, we choose the XLM-R<sub>Base</sub> (Layer number = 12, Hidden dimension = 768, Attention head = 12, 270M params) as our sentence encoder for language-agnostic representations at the post level. We use accuracy and macro-averaged F1 score, as well as class-specific F1 score as the evaluation metrics. Unusually, to conduct five-fold cross-validation on the target dataset in our low-resource settings, we use each fold (about 80 claim posts with propagation threads in the target data) in turn for training, and test on the rest of the dataset. The average runtime for our approach on five-fold cross-validation in one iteration is about 3 hours. The number of total trainable parameters is 1,117,954 for our model. We implement our model with pytorch<sup>8</sup>.
C.1 Effect of Adversarial Perturbation Norm
Figure 5 shows the effect of adversarial perturbation norm on rumor detection performance. The X-axis denotes the value of  $\epsilon$ , where  $\epsilon = 0.0$  in the line means no adversarial augmentation. In general, the adversarial augmentation contributes to the improvements and  $\epsilon \in [1.0,2.0)$  achieves better performances. For the Weibo-COVID19 dataset, our proposed approach ACLR with a smaller adversarial perturbation can still obtain better results but lower than the results with an optimal range of perturbation, while large norms tend to damage the effect of ACLR. In terms of Twitter-COVID19, our method still performs well with a broad range of adversarial perturbations and the performance tends to stabilize as the norm value increases.
$^{6}$ https://twarc-project.readthedocs.io/en/latest/twarc2_en_us/
<sup>7</sup>https://service_account.weibo.com/
pytorch.org
Cross-Domain&Linguual Settings Statistics	Source TWITTER	Target Weibo-COVID19	Source WEIBO	Target Twitter-COVID19
# of events	1154	399	4649	400
# of tree nodes	60409	26687	1956449	406185
# of non-rumors	579	146	2336	148
# of rumors	575	253	2313	252
Avg. time length/tree	389 Hours	248 Hours	1007 Hours	2497 Hours
Avg. depth/tree	11.67	4.31	49.85	143.03
Avg. # of posts/tree	52	67	420	1015
Domain	Open	COVID-19	Open	COVID-19
Language	English	Chinese	Chinese	English
Table 3: Statistics of Datasets in Cross-Domain and Cross-Lingual Settings.
Figure 5: Effect of Adversarial Perturbation Norm  $\epsilon$ .
Figure 6: Effect of trade-off parameter  $\alpha$ .
C.2 Effect of Trade-off Parameter between Classification and Contrastive Objectives
To study the effects of the trade-off hyperparameter in our training paradigm, we conduct ablation analysis under ACLR architecture (Figure 6). We can see that  $\alpha = 0.5$  achieves the best performance while the point where  $\alpha = 0.3$  also has good performance. Looking at the overall trend, the performance fluctuates more or less as the value of  $\alpha$  grows. We conjecture that this is because the supervised contrastive objective, while optimizing
the representation distribution, compromises the mapping relationship with labels. Multitask means optimizing two losses simultaneously. This setting leads to mutual interference between two tasks, which affects the convergence effect. This phenomenon points out the direction for our further research in the future.
C.3 Effect of Target Training Data Size.
Figure 7 shows the effect of target training data size. We randomly choose training data with a certain proportion from target data and use the rest set for evaluation. We use the cross-domain and cross-lingual settings concurrently for model training, the same as the main experiments. Results show that with the decrease of training data size, the performance gradually decreases. Especially for Weibo-COVID19, it will be greatly affected. However, even when only 20 target data are used for training, our model can still achieve more than approximately  $60\%$  and  $65\%$  rumor detection performance (Macro F1 score) on two target data sets. Weibo-COVID19 and Twitter-COVID19 respectively, which further proves ACLR has strong applicability for improving low-resource rumor detection on social media.
C.4 Discussion about Low-Resource Settings
In this section, we evaluate our proposed framework with different source datasets to discuss the low-resource settings in our experiments. Considering the cross-domain and cross-lingual settings in the main experiments, we also conduct an experiment in cross-domain settings. Specifically, for the Weibo-COVID as the target data, we utilize the WEIBO dataset as the source data with rich annotation. In terms of Twitter-COVID19, we set the TWITTER dataset as the source data. Ta
Figure 7: Effect of target training data size.
Target	Weibo-COVID19	Twitter-COVID19
Settings	Acc.	Mac-F1	Acc.	Mac-F1
Cross-D&L	0.873	0.861	0.765	0.686
Cross-D	0.884	0.855	0.737	0.623
Table 4: Rumor detection results of our proposed framework in different low-resource settings. Cross-D&L denotes the cross-domain and cross-lingual settings and Cross-D denotes the cross-domain and monolingual settings.
ble 4 depicted the results in different low-resource settings. It can be seen from the results that our model performs generally better in cross-domain and cross-lingual settings concurrently than that only in cross-domain settings, which demonstrates the key insight to bridge the low-resource gap is to relieve the limitation imposed by the specific language resource dependency besides the specific domain. Our proposed adversarial contrastive learning framework could alleviate the low-resource issue of rumor detection as well as reduce the heavy reliance on datasets annotated with specific domain and language knowledge.
D Future Work
We will explore the following directions in the future:
1. We are going to explore the pre-training method with contrastive learning and then finetune the model with classification loss, which may further improve the performance and stability of the model.
2. Considering that our model has explicitly overcome the restriction of both domain and language usage in different datasets, we plan to evaluate our model on the datasets about
more breaking events in low-resource domains and/or languages by leveraging existing datasets with rich annotation. We believe that our work could provide new guidance for future rumor detection about breaking events on social media.