Leveraging Contrastive Learning and Knowledge Distillation for Incomplete Modality Rumor Detection
Authors: Fan Xu
Pinyun Fu
Qi Huang
Bowei Zou
AiTi Aw
Mingwen Wang
Venue: EMNLP 2023 Findings (2023)
Keywords: Rumor detection
Contrastive learning
Knowledge distillation
Incomplete modality
TL;DR: This paper introduces CLKD-IMRD, a novel framework leveraging contrastive learning and knowledge distillation to address the challenges of incomplete modality in rumor detection.

Abstract: Rumors spread rapidly through online social microblogs at a relatively low cost, causing substantial economic losses and negative consequences in our daily lives. Existing rumor detection models often neglect the underlying semantic coherence between text and image components in multimodal posts, as well as the challenges posed by incomplete modalities in single modal posts, such as missing text or images. This paper presents CLKD-IMRD, a novel framework for Incomplete Modality Rumor Detection. CLKD-IMRD employs Contrastive Learning and Knowledge Distillation to capture the semantic consistency between text and image pairs, while also enhancing model generalization to incomplete modalities within individual posts. Extensive experimental results demonstrate that our CLKD-IMRD outperforms state-of-the-art methods on two English and two Chinese benchmark datasets for rumor detection in social media.

Leveraging Contrastive Learning and Knowledge Distillation for Incomplete Modality Rumor Detection
Fan Xu', Pinyun Fu', Qi Huang', Bowei Zou"; AiTi Aw?, Mingwen Wang'
Jiangxi Normal University
Institute for Infocomm Research, A
(xufan, huangqi, mwwang) (aaraangn(jjiaaaaae @jxnu.edu.cn
fupinyun@ 163.com
grou_bowei, aaiti) | @ir..a-itareeee -ttaasstr.
Abstract
them (Vosoughi al., 2018). Suffering from the low coverage and long delay of rumor detec- tion manually, automatic rumor detection model- 8 are essential. Previous text modality-based ru- mor detection models focused on exploring prop- agation (Lao et al., 2021), user information (Li et al., 2019), and writing styles (Przybyla, 2020; Xu et al., 2020).
Rumors spread rapidly through online social microblogs relatively low cost, causing substantial economic losses and negative con- sequences in our daily lives. Existing rumor detection models often neglect the underlying semantic coherence hetween text and image components multimodal posts, well as the challenges posed by incomplete modalities single modal posts, such as missing text or images. This paper presents CLKD-IMRD, novel framework for Incomplete Modality Ru- mor Detection. CLKD-IMRD employs Con- trastive Learning and Knowledge Distillation to capture the semantic consistency between text and image pairs, while also enhancing model generalization to incomplete modalities within individual posts. Extensive experimen- tal results demonstrate that CLKD-IMRD outperforms state-of-the-art methods on two English and two Chinese benchmark datasets for rumor detection in social media!.
Furthermore, reported in Jin et al. (2017), mi- croblogs with pictures have been found to have 11 times more access than those without pictures, which highlights the importance of multimodal content in rumor detection. Specifically, some studies have investigated the fusion of differen- t modalities, such as images and text, by directly concatenating their representations (Khattar et al., 2019; Singhal et al., 2022). Nevertheless, directly concatenating two different modalities (i.e., tex- tual and visual modalities) may not capture their deep semantic interactions. To address this issue, -r-attention-drivee rumor detection models aim to extract the alignment between the two modalities (Wu et al., 2021; Zheng et al., 2022). However, current multimodal rumor detection models typi- cally overlook the problem of incomplete modali- ties, such the lack of images text in given post, making them less effective in handling such cases.
1 Introduction
Social media platforms like Twitter and Weibo al- low people contribute vast amounts of content the Internet. However, this content is often rife with rumors, which can lead to significant societal problems. For instance, in the first three months of 2020, nearly 6,000 people were hospitalized due to coronavirus misinformation?, while COVID-19 vaccine misinformation and disinformation are timated to cost 50 to 300 million each d
To address the above issues, we propose nov- el approach, Contrastive Learning and Knowledge Distillation for Incomplete Modality Rumor De- tection (CLKD-IMRD). trast learning effectively pulls together represen- tations of the same class while excluding repre- sentations from different classes, and knowledge distillation can effectively handle the incomplete modality cases when debunking rumors. More specifically, we first construct teacher model that consists of multimodal feature extraction, mul- timodal feature fusion, and contrastive learning. Then, we adopt knowledge distillation to construct
Humans generally susceptible to false infor- mation or rumors and may inadvertently spread
fact, supervised con-
Corresponding author.
'Code for all experiments in this paper are availahle "hups://ww ww. hot/news-room/fetuue
bories/detail/iihttingg-- misinformation in the-time-of-covid
one-ccicck-a--a aiii--t----time
"http://asprtracie www..h..oooaa.. ths..oo/technical
resources/rssource/ 1632/covid-11 http///vid----vacceeeeeeeeeeeeee 19-vaccieemmiiinfoomatoon
andddiiinnfrmmatonn-co cowta--oosttttteteeeeeeeeeeeee-e tet---siimtdd-One -00-aat---------000 10003300-miiooo
each-day

---

student model that can handle incomplete modal- ities (i.e., lack of images or texts). Experimen- tal results and visualizations demonstrate that our BLKD-IMRD outperforms state-of tiiee-of-hhe-ae the-art meth- ods on both English and Chinese datasets in rumor detection on social media.
gagement metries (e.g., number of answers, rat- ings given, comments, ratings received, disagree- ment and number of raters), inter-user agreement, typical perspective and expertise, and interaction- s. Yuan et al. (2020) proposed a fake news detec- tion model based on integrating the reputation of the publishers and reposted users. Li et al. (2019) presented a user credit driven multi-task learning- based framework to conduct rumor detection and stance detection simultaneously.
This paper makes three major contributions. (I) We propose novel rumor detection frame- work that integrates the supervised contrastive learning into our teacher network. This frame- work captures the deep semantic interactions a- mong source texts, images, and user comments si- multancously.
Furthermore, the writing style is a crucial factor in rumor detection, there often distinct dif- ferences in the vocabulary and syntax used in mu- mors compared to non-rumors. Specifically, Ru- bin et al. (2015) proposed a rhetorical structure- based framework for news verbification. The rhetorical structure is a wildly used theory in dis- course analysis, which describes how the con- stituent units of a discourse are organized into coherent and complete discourse according to certain relationship. According to the rhetori- cal structure theory, the relationship between dis- course units mostly shows the nucleus-satellite re- lationship. Compared with the author's commu- nicative intention, the discourse unit in the core position is in a relatively important position, and there are different discourse relationships between the core discourse unit and the satellite discourse unit. Potthast et al. (2018) adopted some lexical features (e.g., characters uni-gram, bi-gram and tri-gram, stop words, part of speech, readability value, word frequency, proportion of quoted word- s and external links, number of paragraphs, and average length of a text) for fake news detection. Przybyla (2020) presented a writing style-based fake news detection framework to verify the effec- tiveness of sentiment vocabularies.
(2) We present knowledge distillation driven rumor detection model that can handle incomplete modalities (i.e., lack of images or texts), which is a common phenomenon on social media.
(3) We conduct extensive experiments and vi- sualization to verify the effectiveness of the pro- posed CLKD-IMRD model on the four benchmark rumor corpora compared to many strong baseline models.
2
Related Work
2.1 Single Modality-based Models
Generally, propagation patterns are good indica- tors of rumor detection, because the interaction a- mong different kinds of users (e.g., normal users and opinion leaders) source microblog and it- subsequent reactions can help to detect rumor. As mentioned in (Wu et al., 2015), a rumor was first posted by normal users, and then some opin- ion leaders supported it. Finally was reposted by a large number of normal users. The propa- gation of a normal message, however, was very different from the propagation pattern of a fake news. The normal message was posted by opinion leaders and was reposted directly by many normal users instead. Ma et al. (2017) also proposed propagation path-based model to debunk rumors. Lao et al. (2021) presented a propagation- based rumor detection model to extract the linear tempo- ral sequence and the non-linenn con-linear diffusion structure simultancously.
2.2 Multimodal- Models
Wang al. (2018) presented a GAN-based mul- timodal fake news detection model to adopt VG- G (Visual Geometry Group) to extract visual fea- tures and employ a CNN (Convolutional neural network) to extract text features simultaneously. They also integrated the characteristics of the in- variance of event facilitate the detection of the newly arrived events of fake news. Khattar et al. (2019) proposed a variational auto-encoder- based fake news detection model to capture the shared representation between textual and visu- al modalities. Their novel decoder can utilize
In addition, user information is also good in- sight to rumor detection, because the authorita- tive users are unlikely to publish a rumor, while normal users have a high probability to produce or repost a rumor instead. Specifically, Mukher- jee and Weikum (2015) assessed user credibility based on several factors, including community en-

---

Multimodal Feature Extraction
Multinsdal Feature Fusion
Ourput Layer
eBREARING Cherke in sepedts balding gellcc coercce
R!
x
Z
Toxt Mied
Costacaiion
Teacher Mooill
RS
Leoving
RedNes00
Costactiion
R2
They dunkl wad dipurdd
R
Ad
Casswal
URREAKING
Studem Medel
Charke tn repeck
CNN
balding gpllic cenncc
Authean
ADillas
Sodrass
Losa
apurd poppr
CNN
Figure 1: Framework of CLKD-IMRD. FC: fully connection; R: rumor; N: non-rumor.
the multimodal representations from variation- al auto-eneoder. Zhou et al. (2020) presented a similarity-based fake new detection model to calculate the similarity between multimodal and cross-modal features jointly. In addition, Dhawan et al. (2022) proposed graph iitention-basee fake detection framework to incorporate the inter- action between textual word information and the local visual object from images. Wei al. (2021) presented a graph convolutional networks-based rumor detection model to investigate the reliability of potential relationships in propagation structures through a Bayesian model. Yuan et al. (2019) p- resented a graph-based rumor detection model to combine the encoding of local semantic and glob- al structural information simultaneously. Zheng al. (2022) proposed GaT-based rumor detec- tion model to integrate textual, visual, and social graphs altogether.
rumors with incomplete modalities.
3 Methodology
3.1 Task Formulation
Let's define {pI,P2, pn] as a set of post- s. Each post p3 consists of {4i, 04 G), where I indicates a source text, U; donates an image, and & refers to comment. We approach rumor de- tection as a binary classification task, with goal of learning function f(p:) -> y. where p: repre- sents the given multi-modal post, and y represents the label assigned to the post, where y=1 indicates rumor and y=0 indicates a for non-rumor.
3.2 Framework of CLKD-IMRD
Figure illustrates proposed CLKD-IMRD framework for rumor detection. Specifically, we employ a multimodal feature extraction module obtain representations of the source text, images, and comments from a given post. For the teacher model, the extracted multimodal features are fed into multimodal feature fusion module. During the multimodal feature fusion phase, we adopt vi- sual features to enhance textual features, using the cross-modal joint attention mechanism to obtain enhanced features between textual and visual rep- resentations. Then, in the output layer module, we integrate features from different modalities into a supervised contrastive learning framework. For
Despite the emergence of many multimodal ru- mor detection models in recent years, they often overlook the distinguishing features of samples with the same or different types of labels. Addi- tionally, these models do not account for incom- plete modality situations, where the image may fail to load, which is a common occurrence on the Internet. To address these gaps, we propose a novel framework based on contrastive learning and knowledge distillation to effectively debunk

---

the student model which lacks visual modality, directly concatenate the representation of source text and comments, and adopt knowledge distilla- tion to obtain corresponding classification results.
where "Il" donates concatation operation. refers to the hitth head, and W E Rided) indicates the output of linear transformation.
Similarly, we obtain the multi-head self- attention feature of visual modality as equation 2 as follows.
3.2.1 Teacher Model
The teacher model is composed of three modules: multimodal feature extraction, multimodal feature fusion, and output layer.
Q K ? vd Vi)w
Z -1lh=1sof
(2)
Multimodal Feature Extraction: For textu- al feature extraction, we utilize CNN to obtain the textual representations of source text and com- ments. Given a text & in a post Pi, first ob- tain its representation as o. where of indicates the word embeddings of the j-th word in a text f,, and E Rd where d do- nates the dimension size of the word embedding. Then, the word embedding matrix O is fed in- CNN framework obtain the feature map S {Sa. S2,,,.. S S.LLLk where le is the size of receptive field. Next, we perform max- pooling on to obtain the $" = mar(S') and extract the final representation for the source tex- t tt as Ri concat( - concatenating of different receptive fields. Sim- ilarly, extract the representation for comment C as R: = concat($ St Si $t=))
To extract the co-attention between the textual and visual modalities, we perform a similar self- attention process, replacing R2 to Z R2 to Zj generate Q Kद and Ví. respectively, and finally obtain the enhanced textual features Z with vi- sual features as equation 3.
f2, o2...,,,, of,,)
K Vd W
Z = col
(3)
Next, we perform the second co-attention be- tween Z and Z to obtain the cross-modality fea- ture Z% and Z as equation and 5, respectively.
$/ $ through
P'Kts T V
We utilize ResNet-50 (He et al., 2016) to extrac- the representation for image 14. Specifically, we first extract the output of the second-to-late layer of ResNet-50 and represent it as Vi feeding it in- to fully connected layer to obtain the final visual features as R oWW Vi5) with the same di- mension size the textual features.
2: = (11==10.f
Vi
Wo
(4)
Vd
U.KTT 0 W/ vd
z = -1ll
(5)
Multimodal Feature Fusion: To capture the interaction among different modalities and en- hance cross modal features, we employ co- attention (Lu et al., 2019) mechanism. For both textual and visual modalitics, we first adopt multi- head self-attention (Vaswani et al., 2017) to en- hance the inner feature representation. For in- stance, given a textual feature Ri, we adopt Qi R.W. Ki R.W.K, and V R.W.V to calcu- late matrix Q, K, and V, respectively, where W.O. W.K. and w.Y E dd & linear transformation where H donates the total number of heads. We obtain the multi-head self-attention feature of tex- tual modality as equation as follows.
Finally, we conduct concatenation for the two enhanced features and the initial comment features to obtain the final multimodal features as Z concat Zi, Zi R.)..
Output Layer: Given that supervised con- trast learning (SCL) (Khosla al., 2017) effec- tively pulls together representations of the same class while excluding representations from differ- ent classes, we incorporate the supervised contrast learning function into our total loss for the rumor detection as equation 6. To enhance the robust- ness of the proposed model, we introduce projec- tion gradient descent (PGD) (Madry et al., 2017) into textual embeddings when calculating the gra- dient of textual features in each training iteration and adopt calculate adversarial perturbations. Then, we recalculate the gradient on the updated textual features, repeat this process with m times,
O.KY"
zi -1II
softmar Vd
W/
(1)

---

and adopt spherical space to limit the range of dis- turbance. Finally, we accumulate the above adver- sarial gradient to the original gradient and adopt for parameter adjustment.
This facilitates the training of the student network, which is simplified, possesses lower complexity, and more suitable for deployment in inference scenarios. The ultimate goal is to achieve effec- tive knowledge transfer.
(6) where {0,1} is hyper parameter, CE do- nates cross entropy (equation 7), and SCL indi- cates supervised contrast learning (equation 8).
LTeacer = (1 A)LCE + XLScL
22 KL (
o
(9)
where q' and q" donate the output of teacher and s- tudent network, respectively, a indicates softmax, refers to scale the temperature of the smooth- ness of two distributions. A lower value of will sharpen the distribution, leading to expanded difference between the two distributions. It con- centrates the distillation on the maximum output predicted by the teacher network. On the other hand, a higher value of T will flatten the distri- bution, narrowing the gap between the teacher and student networks. This broader distribution con- centrates the distillation on the entire output range. Then, the total loss of student model is shown in equation 10.
N C M N
1
LCE
logiie
(7)
where C indicates the label type (i.e., rumor and non-rumor), Vi,e donates the true label with class type c, and Úi. refers to the predicted probability with class type
exp( Z1.ZZ
1 LSCL = N
Σ Σ PEP:
log
EdEA, exp( 2. (8) where donates the set of indexes of training sam- ples, P; embodies the set of indexes of positive samples, A, refers to the indexes of contracted samples, Z stands for the normalization, and r indicates temperature parameter control dif- ferent categories.
P
Lttudent = aLKn + LC
(10)
where CE donates cross entropy, and a is a hyper parameter.
3.2.2 Student Model
As mentioned in the Introduction section, exist- ing rumor detection models ignore the incomplete modality problem. It is common for the image to fail to load in a given multimodal post. Due to the lack of visual information, only textual in- formation can be used for rumor detection. To address this issue, we adopt knowledge distilla- tion (Hinton et al., 2015) to perform incomplete rumor detection based our pre-trained teacher model. The motivation behind knowledge distil- lation is leverage the soft labels predicted by the teacher network to guide the learning of the s- tudent network and improve its performance. By minimizing the distance between the soft probabil- ity distributions of the student and teacher models, measured by the KI loss (equation 9), we aim align the predictions of the student model with those of the teacher model. In essence, knowl- edge distillation serves the purpose of incorporat- ing soft targets associated with the teacher net- work, which exhibits complex yet superior predic- tion performance, into the overall Loss function.
4
Experiments
4.1 Datasets and Evaluation Metrics
We utilize four benchmark multimodal corpora: the two Chinese datasets (e.g., Weibo-19 (Song et al., 2019), Weibo-17 (Jin et al., 2017)) and the two English corpus (e.g., Twitter (Boididou et al., 2018), Pheme (Zubiaga et al., 2017)). Each dataset comprises source text and images. The Weibo- 19 and Pheme have comments, while the Weibo- 17 and Twitter don't 't' have comments. Table I presents the statistics of the four benchmark cor- pora. We adopt four popular evaluation metric- s: i.e., accuracy, precision, recall, and F1-Score, to investigate the performance of our proposed framework and other comparing approaches.
Baselines: We take seven bascline models illustrated in Appendix A.I.
Hyper-parameter Settings: Following the ap- proach of existing baseline systems, we divide the datasct into training, validation, and testing set- s using ratio of 7:1:2, respectively. For word

---

Table 1: Corpus statistics. N: Non con-rumors; R: Rumors.
highest performance in all four measures: ac- curacy, precision, recall, and Fl-Score. On the other hand, MVAE demonstrates the poorest per- formance across all four measures on the Wei- bo dataset, which highlights the ineffectiveness of the superficial combination of textual and visu- al modalities in MVAE. In contrast, the incorpo- ration of event information in the EANN model proves beneficial for debunking rumors. Notably, the SAFE model successfully incorporates a deep interaction between textual and visual modalities, resulting in superior performance.
#N
#R
#lmages
#Comments
Weibo-19
877
590
1.467
4,534
Weibo-17
4,749
4,779
9.528
Twitter
6,026
7,898
514
Pheme 1,428
2,018
21,564
embeddings, we employ Word2Vec-style embed- dings as proposed in (Yuan et al., 2019). The number of head H in self-attention is set to 8. We adopt Adam (Kingma and Ba, 2014) to opti- mize our loss function. The learning rate is set to 0.002, and the batch size is set 64. The value of dropout set to 0.6. The in knowledge distilla- tion is set to 5.0. The in supervised contrastives learning is set to 0.5. The length of source text L; and comment Le are set to 50, a is set to 0.7, and À is set to 0.5. The number of m in resistance dis- turbance is set to 3. We perform 5 runs throughout all experiments and report the average results and standard deviation results.
(2) Among the three social graph- base- lines (EBGCN, GLAN, and MFAN), they demon- strate better performance compared to the simpler EANN and MVAE models. Both EBGCN and GLAN achieve comparable performance they incorporate structural information. However, M- FAN, which combines textual, visual, and social graph-based information, outperforms the others in all four measures: accuracy, precision, recall, and FI-Score.
4.2 Results and Discussion
Performance of Knowledge Distillation: CLKD-IMRE involves adopting a multimodal contrastive learning model as the teacher model, and multimodal and incomplete modal models as the student models. Such teacher-student framework allows us to transfer knowledge from the multimodal teacher model to both multimodal and single- modal student models. We explore four student models, each utilizing only cross-entropy loss as the loss function.
Model Comparison: Tables 2 and present the average performance and standard deviation ob- tained from five executions on both the Chinese and English datasets. On the Pheme and Weibo-19 datasets, since we used the same training, valida- tion, and testing splits the baseline systems, directly compare our results with theirs. In addi- tion, we execute the public available source codes from these baselines on the Twitter and Weibo- 17 datasets. Since EBGCN and GLAN constructe a propagation graph by using comments the source posts, we don't 't report their performance on the Weibo-17 and Twitter datasets without com- ments information. It is evident from Tables 2 and 3 that our CLKD-IMRD outperforms other mod- els in terms of accuracy, precision, recall, and F1- Score measures. This highlights the significance of multimodal feature fusion and contrastive learn- ing our approach. While ChatGPT has proven effective various NLP tasks, its performance in rumor detection is not satisfactory. On the Weibo- 17 and Twitter datasets, only short source texts are utilized without comments a supplementation, resulting in poor performance of the ChatGPT on the rumor detection task. Based on the observa- tions from Tables and we can derive the fol- lowing insights:
The model incorporates al- modalities, including source text, visual in- formation, and user comments.
Student-1:
Student-2: The model focuses textual (source text) and visual modalities.
The model relies on the textu- al modality, considering both the source text and comments.
Student-3:
Student-4: The model exclusively relies on the source text modality.
Limited to space, the knowledge distillation re- sults on the Weibo-19 and Pheme datasets are shown in Table 4, which indicate that all stu- dent models exhibit improvement when guided by the teacher model. Even student-4, which only includes the source text modality, demon- strates a 1.0%-1.7% enhancement in accuracy
(1) Among the three multi-modal baselines (EANN, MVAE, and SAFE), SAFE achieves the

---

Table 2: Performance comparison rumor detection models on the two Chinese datasets.
Weibo-19 19
Weibo-17 17
Accuracy
Precision
Recall
FI-Score
Accuracy
Precisson
Recall
FI-Score
EANN
80.96:12.22
80.19:2.337
79.86:2..6
79.8742.440
82.83:2..4
82.76+2.42
82.6842.58
82.71+2.43
MVAB
71,67:+0.89
70.52::0.95
70.21+1.01
70.34:1.98
81,23:1.225
84,54:1.75
74.68:11.32
79,94=1.58
SAFE
84.95:0.885
84.98:10.82
84,9520.91
84.9620.86
73.56:1143
16.54=1.33
74.95=1.31
74.80=1.46
EBGCN
83.14:42.01
85.46:12.12
11.76::1.54
11.45::1.74
GLAN
82.44:42.02
82.45:4:2:26
80.86:t1.71
81.26:t 1.93
MPAN
88.95:1144
88.91+11600
88. 18.13:11668
88.33:11.33
88.23+1.83
88.56:1191
88.07
88.
1.73
ChatGPT
29.83=20
28.27
28.95::0
28. 28.52:00
0::0
0+:0
0+:0
0::0
CLKD-IMRD
93.36-F1.12
93.38 + 1.23
92.91 + 1,28
93.07 + 1,20
91.25
1.01
91.29 + 1.32
91.34 1,33
91.14 1.71
Table 3: Performance comparison of rumor detection models on the two English datasets.
Twitter
Pheme
Accuracy
Precision
Recall
FI-Score
Accuracy
Precision
Recall
FI-Score
EANN
71.52++1.99 1.96
2.
00.6543.23
2.66
77.13--0.96
1.077-+1007
70.07-2..19 2.19
70.44-+1 1.69
MVAE
79.9140.996 84.884 1.23
71.4811.10
71.57+1.25
77.62:0064
73.49-00881
72,25=0..00
72.77=0.81
SAFE
76.49+11.24 8..18+1.62
79.50++101
79.68+00.98
81.49-10.84
79.88++1.22
79.50+0.81
79.68+0..70
EBGCN
82.99-10..6
8-.31
79.29-+0.71
79.82-+0.64 0.64
GLAN
83.32:11.64 .64
81,25:42.06
3..13:3.26
78.51
MFAN
86.08+1.12
85.03+11.67
+1,9
84.36+11644
88.73-40.83
87.07+1.441
85.61 +-66
86.16-11.04
ChatGPT
0=0
01:0
0::0
0::0
34.29:1:0
24.26:t:0
26.94:1.0
25.53:1:0
CLKD-IMRD
88.01:0.911
88.15:+1.03
84.64:1..65
86.53:10.97
89.09:10.47
87.46:10.96
85.86:10.16
86.58::0.41
and F1-Score measures. Similar improvements are observed in the other three student model- (student-1, student-2, and student-3). Among these, student-1, utilizing all modalities (source text, visual information, and user comments), achieves the best performance across all four mea- sures. Generally, student-2 outperforms student-3 due to its incorporation of both textual and visual modalities, while student-3 relies solely on textual information.
modality in identifying rumors. 2) Both images and comments contribute to debunking rumors, evidenced by their absence leading to decline in performance. 3) The integration of supervised contrastive learning enhances the model's ability to distinguish between positive and negative sam- ples in the corpora, which positively impacts the performance of the model. 4) The inclusion of
projection gradient descent during the adversarial perturbations training phase improves the robust- of our proposed model.
Ablation Study: Limited to space, Table 5 presents the performance of ablation analysis on the Weibo-19 and Pheme datasets, where we ex- amine the impact of various components by con- sidering five cases:
Impact of Co-attention Settings: We further analyze the performance comparison with differ- ent number of co-attention as illustrated in Ap- pendix A.2.
w/o text: We exclude the use of source text.
4.3 Impact of Number of Comments
We omit the utilization of image information.
w/o image:
We further analyze the performance of different comment scenarios, considering the following six
w/o comment: comments.
We disregard the inclusion of
0% comment: No comments are used in this case.
w/o contrastive learning: We eliminate the application of contrastive learning.
only the first comment: Only the first com-
w/o projection gradient descent: We do not
ment is considered.
employ projection gradient descent.
Based on the findings in Table several conclu- sions can be drawn. 1) The source text plays cru- cial role in rumor detection. The performance sig- nificantly deteriorates when the source text is ex- cluded, underscoring the importance the textual
20% % comments: We include 20% of the comments in time-sequential manner.
50% comments: We include 50% of the
comments in a time-sequential manner.

---

Table 4: Performance comparison of knowledge distillation on Weibo- 19 and Pheme. Performance improvement is represented by numbers in parentheses with symbol.
Weiho-19
Accuracy
Precision
Recall
FI Score
Accuracy
Precision
Recail
FI Score
Teacher
94.24
94.07
93,95
94.01
89.61
$8.49
85,92
87.06
Student: 1 initial
93.22
93.33
92.55
92.90
88.31
86.72
84,49
83,48
Studens-1 after
94 24,24411.00
94.67(11
93.4070.855
9393011.5 185)
88 53(10.52)
(7,39(10667)
K5.TI(00.22)
86.13(10.65)
Student-2 Initial
91
90:99
91.67
91.27
86.49
84.63
81,91
83.08
Student-2 2 after KD
93.22(11.69)
93.67(12.68)
92.28(10.61)
92,85("
88.31(11882)
86,23(11.60)
85,26(13.35)
85.72(12.64)
Student-3
92.20
92.33
91.82
87.01
84.56
K3.86
84.18
3 alter KD
93. 1..02)
83.19(10.86)
92.69(11.27
91,922"0.100 (0.10)
B8. 31.3111130
86,91(12.55)
10.23(10.37)
13.223
Studeai-1
90.85
91.85
89.33
90.23
86.23
83.65
82.76
83.18
Student-4
12.3)))
92.33(10:88
91442(209)
9181(1138)
87.79(11.5)
56.05(13.00
(5.08/10.32)
84.57(F1.39)
Table 5: Ablation study.
Welbo-19
Pheme
Accuracy
Precision
Recall
FI-Score
Accuracy
Precision
Recall
FI-Score
CLKD-MRD
94.24
94.07
93.95
94.01
89.61
88.49
85.92
87.06
w/o text
81.02
80.36
80.00
80.17
77.14
72.76
68.56
69.89
w/o image
92.20
92.49
91.29
91.79
88.05
86.30
84.30
85,20
w/o comment
92.54
93.36
91.30
92.08
86.75
84,43
83.12
83.73
w/o contrastive
learning
93.22
93.33
92.55
92,90
88.31
86.72
84,49
85.48
w/o projection
gradient descent
92.54
92.17
92.39
92.27
87.27
84.60
84.78
84.79
Table 6: Performance comparison with different num- ber of comments.
Avcuracy
Precision
Recall
Fl Score
Weibo-19
0% commentt
92.34
93.36
91.30
92.08
only the
first comment
94.24
94.07
93.95
94.01
comments
93.56
92.97
50 % comments
92.88
92.37
93.08
92.67
comments
92.88
92.91
92.58
all commennis
93.22
93.33
92.55
92,90
Pheme
0% commennts
86.75
84.43
83.12
83.73
only the first
89.61
88.49
85.92
87.06
20% comments
87.27
84.71
82.53
84.62
&
87.01
85.14
82.79
83.82
Figure 2: T-SNE visualization the Weibo-19
ND & comments
88002
87.4
13.14
84.74
85,50
Visualization Studies: Figures 2 and dis- play the T-SNE visualization of the test data from Weibo-19 19 and Pheme, respectively. The visualiza- tions clearly depict the successful classification of most samples into distinct groups, demonstrating the effectiveness and strong representation capa- bility of our proposed model.
80% comments:
We include 80% of the comments in a sime-sequential manner.
all comments: All comments are included,
Limited to space, the impact of number of com- ments results on the Weibo-19 and Pheme are shown in Table 6. Based on the findings in Table 6, we can draw the conclusion that increasing the number of comments does not contribute signifi- cantly to debunking rumors. In fact, as the number of comments increases, the introduction of noise becomes more prominent. Interestingly, the first comment proves to be more valuable in the context of rumor detection, as it carries more relevant in- formation for distinguishing between rumors and non-rumors.
Figures 4 and 5 showcase the attention visual- ization samples with the label "non-rumor" and "rumor" from the Weibo-19 and Pheme, respec- tively, which provides insights into the interaction between textual and visual information, and how the enhanced features contribute to debunking ru- mors. In Figure 4, the words "cat" and "dog" high- lighted in red demonstrate high attention weight- s and align well with specific regions in the cor- responding image. This accurate alignment con-

---

5 Conclusion
In this paper, propose detection frame- work that combines supervised contrastive learn- ing and knowledge distillation. Our framework leverages hierarchical co-attention to enhance the representation of textual (source text and com- ments) and visual modalities, enabling them to complement each other effectively. The utilization of contrastive learning has proven to be success- ful in debunking rumors. Additionally, knowledge distillation has demonstrated its efficacy in han- dling incomplete modalities for rumor detection. Moving forward, our future work aims integrate graph structures, such as social graphs, into our proposed framework for further improvement.
0.-
02
00
03
02
06
1.0
Figure 3: T-SNE visualization on the Pheme.
Acknowledgments
The authors would like to thank anonymous re- viewers for their insightful comments on this pa- per. This research was supported by the National Natural Science Foundation of China under Grant 62162031, the Natural Science Fund project in Jiangxi province under Grant 20224ACB202010, and the National Natural Science Foundation of China under Grant 62266023.
MA TEbif 30 PE ftl. Disabled cat die from hunger strike after its playmate dog dies from cancer. Label: Non-rumor.
Figure 4: Attention visualization on the Weibo-19,
Limitations
tributes the prediction of the sample as non- rumor. In contrast, in Figure 5, the words "sus- pect" and "MartinPlace" fail to align with their re- spective image regions, indicating poor alignmen- and predicting the sample as a rumor correctly. These observations highlight the deep semantic in- teraction between the textual and visual modalities within proposed model.
(1) Limited generalizability: Our experiments were conducted on specific datasets (Chinese Wei- bo and English Pheme/Twitter) and may not fully represent the characteristics of other rumor detec- tion scenarios platforms. Generalizability to d- ifferent datasets and languages needs to be further explored.
(2) Absence of real-time evaluation: Our eval- uation primarily focused on offline performance measures, and we did not consider real-time or dy- namic evaluation scenarios. Future work should investigate the model's performance in real-time rumor detection settings.
Ethics Statement
The benchmark datasets utilized in the project pri- marily reffect the culture of the English-speaking and Chinese-speaking populace. Socio-economic biases may exist in the public and widely-used datasets, and models trained on these datasets may propagate this biases.
Police take down suspect MartinPlace. Unelear whether related to siege. Label: rumor.
Figure 5: Attention visualization on the Pheme.

---

References
Jing Ma. Wei Gao, and Kam-Fai Wong. 2017. Detect rumors in microblog posts using propagation struc- ture via kernel learning. In Proceedings of the An- nual Meeting of the Association for Computational Linguistics (ACL'17), pages 708-717. Vancouver, Canada.
Christina Boididou, Symeon Papadopoulos, Markos Zampoglou, Lazaros Apostolidis, Olga Pa- padopoulon, and Yiannis Kompatsiaris. 2018. Detection and visualization of misleading content on twitter. International Journal of Multimedia Information Retrieval, 7(1):71 86.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2017. Towards deep learning models resistant to adversar- ial attacks. arXiv preprint arXiv: 170.v 06083.
Mudit Dhawan, Shakshi Sharma, Aditya Kadam, Ra- jesh Sharma, and Ponnurangam Kumaraguru. 2022. Game-on: graph attention network based multi- modal fusion for fake news detection. arXiv preprint arXiv 2202. 12478v2.
Subhabrata Mukherjee and Gerhard Weikum. 2015. Leveraging joint interactions for credibility analy- sis in news communities. ACM International Conference on Infomlation and Knowledge Management (CIKM' 15), pages 353- 362. Melbourne, Australia.
Proceedings of the
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recog- nition. In Proceedings of the 19th IEEE Internation- al Conference Data Mining (CVPR'16) 16), pages 770-778. Las vegas, USA.
Martin Potthast, Johannes Kiesel, Kevin Reinartz, Janek Bevendorff, and Benno Stein. 2018. A stylo- metric inquiry into hyperpartisan and fake news. In Proceedings of the 56th Anmual Meeting of the As- sociation for Computational Linguistics (ACL'18), pages 231-240. Melbourne, Australia.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint 1503.025
Zhiwei Jin, Juan Cao, Yongdong Zhang, Jianshe Zhou, and Qi Tian. 2017. Novel visual and statistical im- age features for microblogs news verication. IEEE Transactions on Multimedia, 19(3):598-00 668..
Piotr Przybyla. 2020. Capturing the style of fake news. In Proceedings of the Thirty-Fourth AAAI Con- ference on Artificial Intelligence (AAAI'20), pages 490-497. New York, USA.
Dhruv Khattar, Jaipal Singh Goud, Manish Gupta, and
Vasudeva Varma. 2019. Mvae: multimodal varia- tional autoencoder for fake news detection. In Pro- ceedings of the International World Wide Web Con- ferences ( (WWW'99) pages 2921- San Fran- cisco, USA.
Victoria Rubin, Niall Conroy, and Yimin Chen. 20155 Towards news verification: Deception detection methods for news discourse. In Proceedings of the Hawati International Conference on System Sci- ences (HICSS48) Symposium on Rapid Screening Technologies, Deception Detection and Credibility Assessment Symposium, pages 1-11. Hawaii, USA.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2017. Su- pervised contrastive learning. In Proceedings of the 34th International Conference on Neural Infor- mation Processing Systems (NIPS'17), 17), pages 5998- 6008. Online.
Shivangi Singhal. Tanisha Pandey, Saksham Mrig. Rajiv Ratn Shah, and Ponnurangam Kumaraguru. 2022. Leveraging intra and inter modality relation- ship for multimodal fake news detection. In Pro- ceedings of the Web Conference (WWW'22), pages 726-734. Online.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6900.
Changhe Song, Cheng Yang, Huimin Chen, Cunchao Tu, Zhiyuan Liu, and Maosong Sun. 2019. Ced: Credible early detection of social media rumors. IEEF Transactions on Knowledge and Data Engi- neering, 33(8):3035-3047
An Lao, Chongyang Shi, and Yayi Yang. 2021. Ru- mor detection with field of linear and non-linear propagation. In Proceedings of the Web Conference (WWW211), pages 3178-3187. Ljubljana, Slovenia.
Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is al- you need. In Proceedings of the 31st Interna- tional Conference on Neural Information Processing Systems (NIPS'17), pages 5998-6008. Long Beach California, USA.
Quanzhi Li, Qiong Zhang, and Luo Si. 2019. Rumor detection by exploiting user credibility information, attention and multi-task learning. pages 1173-1179, Florence, Italy.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lec. 2019. Vilbert: Pretraining task-agnostic visi- olinguistic representations for andio--nd--anuuage tasks. arXiv preprint arXiv: 1908, 02265.
Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018. The spread of true and false news online. Science, 359()) 1151

---

Yaqing Wang, Fenglong Ma, Zhiwei Jin, Ye Yuan, Guangxu Xun, Kishlay Jha, Lu Su, and Jing Gao. 2018. Eann: Event adversarial neural networks for multi-modal fake news detection. In Proceedings of the 24th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD'18) 18), pages 849- 857. London, UK.
Lingwei Wei, Dou Hu, Wei Zhou, Zhaojuan Yue, and Songlin Hu. 2021. Towards propagation uncertain- ty: Edge-enhanced bayesian graph convolutional networks for rumor detection. arXiv preprint arX- iv:2107.11932
Ke Wu, Song Yang, and Kenny Q. Zhu. 2015. False ru- mors detection on sina weibo by propagation struc- tures. Proceedings of the IEEE International Conference on Data Engineering (ICDE'15) pages 651-662. Scoul, Korea.
Yang Wu, Pengwei Zhan, Yunjian Zhang. Liming Wang, and Zhen Xu. 2021. Multimodal fusion with co-attention networks for fake news detection. In Proceedings of the Findings of the Association for Computational Linguistics (ACL-UCNLLP'1I), pages 2560-2569. Online.
Fan Xu, S. Sheng Victor, and Mingwen Wang. 2020. Near real-time topic-driven rumor detection source microblogs. Knowledge-Baud BBased Systems, 207:1-9.
Chunyuan Yuan, Qianwen Ma. Wei Zhou, Jizhong Han, and Songlin Hu. 2019. Jointly embedding the local and global relations heterogeneous graph for rumor detection. In Proceedings of the 19th IEEE International Conference on Data Mining (ICD- M'I9), pages 705- Beijing. China.
Chunyuan Yuan, Qianwen Ma. Wei Zhou, Jizhong Han, and Songlin Hu. 2020, Early detection of fake by utilizing the credibility of news, publisher- 8, and users based on weakly supervised learning. In Proceedings of the 28th International Conference on Computational Linguistics (COLING 20), pages 5444-5454. Online.
Jiqqi Zheng, Xi Zhang, Sanchuan Guo, Quan Wang, Wenyu Zang, and Yongdong Zhang. 2022. M- fan: Multi-modal feature-enhanced attention net- works for rumor detection. In Proceedings of the 31st International Joint Conference on Artificial In- telligence (IJCAI'22), pages 2413-2419. Vienna, Austria.
Xinyi Zhou, Jindi Wu, and Reza Zafarani. 2020. Safe: Similarity-awaan multi-modal fake news dectection. arXiv preprint arXiv:2003.04900
Arkaitz Zubiaga, Maria Liakata, and Rob Procter. 2017. Exploiting context for rumour detection in so- cial media. In Proceedings of the International Con- ference on Social Informatics (SocInfo' 17), pages 109-123. Oxford, UK.

---

A Appendix
Table 7: Performance comparison with different num-
ber of co-attention; co-att donates co-attention.
A.1 Baselines
Accuracy
Precision
Recall
FI -core
To investigate the performance of our proposed
Weibo-19
CLKD-IMRD model, we will perform comparison
Zero co-att
92.54
92.04
92.66
92.31
One co-att
93.22
93.06
92.82
92.94
studies on following approaches.
Two co-alt
93.90
93.77
93.53
93.64
EANN (Wang et al., 2018): A GAN-based mul-
Three co-att
94.24
94.07
93.95
94.01
timodal model that adopts characteristics of the in-
Pheme
7ero co-att
86.49
83.31
85.79
84.33
variance of an event to facilitate the detection of
One co-att
88.57
87.58
84.15
85.60
the newly arrived events fake news.
Two co-all
88.83
87.80
84,59
85.96
MVAE (Khattar et al., 2019): A variational auto nuto--eccodr-bbee based model that captures the shared
Three co-att
89.61
88.49
85.92
87.06
representation between textual and visual modali- ties.
One Co-attention: Here, only the first co- attention is employed.
SAFE (Zhou al., 2020): similarity-aware multimodal model that debunks fake news from the similarity between multimodal and cross- modal features jointly.
Two Co-attention: In this case, two text- visual co-attention operations are conducted. The enhanced textual-visual representation is then concatenated with the comment repre- sentation.
EBGCN (Wei et al.. 2021): An edge- enhanced bayesian graph convolutional networks- based model that investigates the reliability of po- tential relationships in propagation structures.
Three Co-attention: This case involves the adoption of all three co-attention operations. The enhanced textual and visual representa- tions are concatenated with the comment rep- resentations.
GLAN (Yuan et al., 2019): An integration of local semantic and global structural information- based model that debunks rumor.
MEAN (Zheng al., 2022): A feature- enhanced attention networks-based multimodal model that combines textual, visual, and social graphs to enhances graph topology and neighbor- hood aggregation processes when detecting rumor. ChatGPT 4, A popular application showcasing the capabilities of the GPT language model is our baseline model. Since ChatGPT cannot receive image modality, adopt the source text and the first comment the input of ChatGPT, along with a question "judge a rumor or not" to obtain the response, and map the results labels (i.e.g. "yes" to rumor, "no" to non-rumor, "unable to judge" to none).
From Table 7, we observe that the perfor- mance improves with an increase in the num- ber of co-attentions. Specifically, the Zero co- attention case demonstrates the lowest perfor- mance across all three measures (accuracy, preci- sion, Fl-Score), which indicates the importance of capturing the deep interaction between textual and visual modalities through co-attentions. With the addition of one co-attention, provement in performance as the enhanced textu- al representation aids in debunking rumors. As expected, the best performance is achieved when both the enhanced textual and visual representa- tions are utilized, as evidenced by their superior results across all four measures.
observe im-
A.2 Impact of Co-attention Settings
Limited to space, Table 7 lists the performance comparison with different number of co-attention on the Weibo-19 and Pheme datasets. We consider four cases follows.
Zero Co-attention: In this case, no co- attention is used. The representations of the source text, visual images, and comments directly concatenated.
"httss//opemai.com/bbgggggggg


Reviews:
[{"review_id": "odJR0vJ18R", "reviewer": "Reviewer_DUWX", "summary": "", "questions": "1) This paper aims to address the incomplete rumor detection, where there is perhaps lack of image or text information in given posts. However, the learning of student model is dependent on the teacher model that will require both complete image and text modalities.\n2) In Table 2, the results of the proposed method and MFAN are comparable. It would be good to explain more on the case.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful comments. We answered your comments as per below.\n \n1.\tThis paper aims to address the incomplete rumor detection, where there is perhaps lack of image or text information in given posts. However, the learning of student model is dependent on the teacher model that will require both complete image and text modalities.\n\n[Answer] In this work, we focus on the multi-modal rumor detection task. It is indeed true that our teacher model relies on complete modalities, encompassing both images and texts during the training phase. The teacher model's purpose is to construct a unified semantic space that encapsulates the semantic information from both image and text modalities. As indicated in our paper, it outperforms the seven baseline systems, showcasing its efficacy. However, the crucial distinction lies in the capabilities of our student model. While the teacher model necessitates complete modalities for training, the student model is designed to handle incomplete modalities during the inference phase. This means that it can effectively perform rumor detection even when presented with posts containing missing text or images, making it highly adaptable to real-world scenarios where information may be incomplete or unreliable.\n\nFurthermore, our experimental results clearly demonstrate the versatility and effectiveness of our student models across different scenarios. Specifically, our four student models, including student-1, student-2, student-3, and student-4, exhibit exceptional performance on the Weibo dataset. Notably, the two student models that leverage both text and image modalities, i.e., student-1 and student-2, excel on the Pheme dataset. Importantly, our two student models that solely utilize text modality, student-3 and student-4, achieve comparable performance with the state-of-the-art MFAN model on the Pheme dataset, underscoring the robustness of our approach.\n\nAdditionally, we have extended our evaluation to include two new, larger multi-modal datasets, Weibo-2017 (Jin et al., 2017) and Twitter  (Boididou et al., 2018), as indicated in Table 2 in our manuscript. These new datasets serve as a validation of the scalability and generalizability of our approach. We intend to provide detailed performance comparisons with the other six baseline systems in the camera-ready version of our manuscript, should it be accepted.\n\nTable 1: Statistics of the two new datasets. N: Non-rumors; R: Rumors.\n|            | #N   | #R   | #Images |\n|------------|------|------|---------|\n| Weibo-2017 | 4749 | 4779 | 9528    |\n| Twitter    | 6026 | 7898 | 514     |\n\nTable 2: Performance on the two new datasets.\n| Model           | Accuracy | Precision | Recall | F1-Score |\n|-----------------|----------|-----------|--------|----------|\n|      |          |      Weibo-2017      |        |          |\n| CLKD-IMRD(ours) | 92.94    | 92.99     | 93.03  | 92.94    |\n| MFAN            | 90.23    | 90.56     | 90.07  | 90.17    |\n|          |          |    Twitter       |        |          |\n| CLKD-IMRD(ours) | 91.09    | 92.14     | 88.95  | 90.15    |\n| MFAN            | 89.57    | 89.02     | 88.63  | 88.82    |\n\n2.\tIn Table 2, the results of the proposed method and MFAN are comparable. It would be good to explain more on the case.\n\n[Answer] Indeed, as highlighted in our paper, our model demonstrates superior performance over the MFAN model on the Weibo dataset, with a substantial margin in accuracy (4.3%) and F1-score (4.7%). This clear advantage can be attributed to the effectiveness of our approach in capturing and leveraging multi-modal information to make accurate predictions in the context of the Weibo dataset. While, on the Pheme dataset, our model achieves results that are comparable to those of the MFAN model, but with a more stable variance. \n\nFurthermore, it is worth noting that while the MFAN model is a complex architecture that integrates textual, visual, and intricate social graph information, our model doesn't rely on such complicated social graph structures. This simplicity in model design contributes to the robustness of our approach and its adaptability to different datasets and scenarios.\n\nTo provide a concrete example of our model's efficacy, we include one more case study in Figure 1 below (we cannot upload the image here) in our paper. In this instance, the MFAN model made an incorrect prediction, labeling a post as \"non-rumor,\" whereas our model, when integrating user comments, correctly predicted it as a rumor. This example underscores our model's ability to effectively harness multi-modal information to make accurate predictions.\n\nWe appreciate the reviewer's suggestion and will ensure that these explanations are included in our revised manuscript to provide a more comprehensive understanding of the comparative results presented in Table 2.\n\nText:[I'm not the hero，I'm a tool]The Red Cross Society of China has sent a congratulatory message: Warmly congratulate Guo Meimei on her appointment as a cover person of Time magazine in the United States!\n\nComments：The media likes to hype up such topics, most of which are fake news.\n\nLabel：Rumor\n\nFigure 1: One more case study", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The work presents a rumor detection framework that relies on the supervised contrastive learning and teacher network. The framework can capture semantic interactions among source texts, images, and user comments.\n2) This paper presents a knowledge distillation driven rumor detection model that can probably handle incomplete modalities (i.e., lack of images or texts).", "reasons_to_reject": "1) Multimodal rumor detection is not new, and this work focuses on incomplete modality case, where it may be lack of image or text information in given posts. Thus it may lead to incremental contribution.\n2) The proposed method was tested on only 2 small datasets, and it's not clear if it can be generalizable to different runor detection tasks.\n", "questions_to_authors": "1) This paper aims to address the incomplete rumor detection, where there is perhaps lack of image or text information in given posts. However, the learning of student model is dependent on the teacher model that will require both complete image and text modalities.\n2) In Table 2, the results of the proposed method and MFAN are comparable. It would be good to explain more on the case.", "paper_topic_and_main_contributions": "This paper is about incomplete rumor detection task on the social media platforms (e.g., Twitter or Weibo). It presents a framework of using contrastive learning and knowledge distillation for the incomplete modality rumor detection problem. The proposed method can capture the semantic consistency between text and image pairs, while it can also enhance model performance to incomplete modality cases of real posts. This paper was well written and is easy to follow.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "Line 065: \"Infact\"\n", "Justification_for_Ethical_Concerns": "NA"}, "extra_text": {}}, {"review_id": "ycghNWLBzC", "reviewer": "Reviewer_Ytxq", "summary": "", "questions": "- Why did you not take into account the case of missing text information? Would the model need modifications?\n- Is there a reason for not using pre-trained embeddings?\n- Did you take the results directly from the original papers for the baseline models or did you re-implement them?\n- How would you compare this method to the baseline models in terms of efficiency? Is this method scalable?\n- In page 8 (visualisation studies): how did you choose these examples? \nThe analysis on number of comments is more interesting and objective and I suggest to move it to the main body of the paper (see reasons to accept).", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful comments. We answered your comments as per below.\n\n1.\tWhy did you not take into account the case of missing text information? Would the model need modifications?\n\n[Answer] In the context of social media platforms such as Twitter and Weibo, textual information is typically a fundamental component of posts due to constraints like character limitations. Given this common occurrence of text in social media data, our primary focus was on multi-modal rumor detection where both text and image modalities are present. Therefore, we did not explicitly address the case of missing text information in our current work.\n\nHowever, it's worth noting that handling the scenario of missing text can indeed be a valuable extension of our model. This specific task, often referred to as deepfake detection, involves verifying the authenticity of an image or video when textual information is absent.\n\nThe good news is that our model is inherently flexible and modular in design. Should one wish to adapt it to handle cases of missing text, the required modification is relatively straightforward. The only necessary adjustment would be to deactivate or remove the text encoding module within our framework while retaining the image-related components. This adaptation would allow our model to effectively process and analyze the available image data for rumor detection when textual information is missing. We appreciate the reviewer's feedback and will consider incorporating this clarification into our revised manuscript.\n  \n2.\tIs there a reason for not using pre-trained embeddings?\n\n[Answer] Upon careful evaluation, we observed that the word2vec embeddings yielded slightly better performance compared to the BERT embeddings in the context of our multi-modal rumor detection task. The decision to prioritize word2vec over BERT was based on the specific requirements and characteristics of our dataset and the task at hand. It's important to note that the choice of embeddings can be highly dataset-dependent, and in our case, word2vec embeddings proved to be more suitable for the task. Nevertheless, we acknowledge that the use of different embeddings or fine-tuning techniques may yield varying results on other datasets or tasks. Our approach is designed to be flexible, allowing for the incorporation of different embeddings or modifications based on the specific needs of the problem.\n \n\n3.\tDid you take the results directly from the original papers for the baseline models or did you re-implement them?\n\n[Answer] We have pointed out it in the line 417-420 of Page 6 in the previous manuscript. Since we used the same training, validation, and testing splits as the baseline systems, we directly compare our results with theirs.\n\n4.\tHow would you compare this method to the baseline models in terms of efficiency? Is this method scalable?\n\n[Answer] We have conducted an initial efficiency comparison with the state-of-the-art MFAN model on the two new larger datasets, Weibo-2017 (Jin et al., 2017) and Twitter  (Boididou et al., 2018), and have considered various aspects, including training time, testing time, and parameter size. The results are presented in Table 2, as shown below. It is evident that our model outperforms the SOTA MFAN model in terms of efficiency, as it has fewer parameters, shorter training times, and faster testing times. In addition, we provide the performance comparison with the SOTA MFAN model on the two new larger multi-modal datasets (e.g., Weibo-2017 and Twitter) as shown in Table 3 below. We can conclude that our mode also works on the two new larger multi-modal datasets. We will provide other performance data for the left six baseline systems in the camera-ready version if the manuscript is accepted.\n\nOur approach exhibits scalability on two fronts:\n\n(1) Our model is designed with a modular architecture, allowing for flexibility and adaptability to different requirements. It comprises pluggable modules, including text encoding, image encoding, supervised contrast learning, attention mechanisms, and more. These modules can be selectively removed or adjusted as needed. For instance, when facing scenarios with missing images or text, the corresponding encoding modules can be deactivated. This adaptability not only enhances efficiency but also broadens the model's applicability to various tasks. For instance, by focusing solely on image modality, our model can be extended to tackle deepfake detection, which verifies the authenticity of images or videos.\n\n(2) Our model is not limited to a specific literary form and can be applied to different types of content, including fake news or rumors.\n\nTable 1: Statistics of the two new datasets. N: Non-rumors; R: Rumors.\n|            | #N   | #R   | #Images |\n|------------|------|------|---------|\n| Weibo-2017 | 4749 | 4779 | 9528    |\n| Twitter    | 6026 | 7898 | 514     |\n\nTable 2: Efficiency on the two previous datasets.\n|       |                |Weibo-2019|                |\n|-----------------|----------------|--------------|----------------|\n| Model           | Training time (in seconds)  | Testing time (in seconds) | Parameter size  (in MB) |\n| CLKD-IMRD(ours) | 1510.89        | 3.86         | 6.56           |\n| MFAN            | 1774.98        | 4.03         | 10.5           |\n|         |                |   Pheme           |                |\n| CLKD-IMRD(ours) | 1554.33        | 2.16         | 6.56           |\n| MFAN            | 1898.07        | 2.51         | 10.5           |\n\nTable 3: Performance on the two new datasets.\n| Model           | Accuracy | Precision | Recall | F1-Score |\n|-----------------|----------|-----------|--------|----------|\n|      |          |      Weibo-2017      |        |          |\n| CLKD-IMRD(ours) | 92.94    | 92.99     | 93.03  | 92.94    |\n| MFAN            | 90.23    | 90.56     | 90.07  | 90.17    |\n|         |          |   Twitter         |        |          |\n| CLKD-IMRD(ours) | 91.09    | 92.14     | 88.95  | 90.15    |\n| MFAN            | 89.57    | 89.02     | 88.63  | 88.82    |\n\n5.\tIn page 8 (visualisation studies): how did you choose these examples? The analysis on number of comments is more interesting and objective and I suggest to move it to the main body of the paper (see reasons to accept).\n\n[Answer] Our intention in choosing these specific examples was to illustrate the potential malalignment between an image and its associated text, as this is a common scenario indicative of rumor creation, where individuals may modify certain areas of an image or alter specific words in a text to propagate false information.\n\nWe acknowledge the reviewer's suggestion to place more emphasis on the analysis of the number of comments, which is indeed a more objective and informative aspect of our study. We will move the analysis comments to the main body of the paper if the manuscript is accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Good motivation and well-conducted experiments. They compare their results to a range of baseline models.\n- They tackle the problem of missing modality. This is not only valuable for rumor detection but for multimodal classification of social media posts in general.\n- The paper present different types of analysis to shed light on the benefits of their approach. For instance, they perform an ablation study, a study using different versions of the student model, and a study on the effect of using different number of comments.\n- Interesting findings, for instance, they show that using more comments does not necessarily improve performance. This is interesting given the idea of \"more data is always better\". This is also opposite to findings in multimodal social media classification: Xu, Chunpu, and Jing Li. \"Borrowing Human Senses: Comment-Aware Self-Training for Social Media Multimodal Classification.\" arXiv preprint arXiv:2303.15016 (2023).", "reasons_to_reject": "- Experiments are only done on two datasets for binary classification. Since you are presenting a new method it is worth applying the models to a range of benchmark datasets.\n- Some choices are not justified/not clear. See questions for the authors.", "questions_to_authors": "- Why did you not take into account the case of missing text information? Would the model need modifications?\n- Is there a reason for not using pre-trained embeddings?\n- Did you take the results directly from the original papers for the baseline models or did you re-implement them?\n- How would you compare this method to the baseline models in terms of efficiency? Is this method scalable?\n- In page 8 (visualisation studies): how did you choose these examples? \nThe analysis on number of comments is more interesting and objective and I suggest to move it to the main body of the paper (see reasons to accept).", "paper_topic_and_main_contributions": "This paper presents a model for rumor detection (binary classification) using supervised contrastive learning and knowledge distillation. They take into account multimodal cases (text and image pairs) as well as text-only examples. They perform experiments on two datasets: Twitter and Weibo where each instance has an image, text, and comments. \n\nThey follow a teacher-student framework:\n- The teacher model consists of three components: multimodal feature extraction, multimodal feature fusion, and output layer. The model takes word2vec-style embeddings which are fed into a CNN framework to obtain feature maps corresponding to the main text and comments. They follow the same approach to encode images using Resnet-50. Then, they use co-attention layers to capture image interactions. They train the model with a combination of cross-entropy loss and supervised contrastive loss.\n\nThe student model is used for addressing text-only cases. They use knowledge distillation where they leverage the soft labels predicted by the teacher network to guide the learning of the student network and improve its performance. The model is trained with a combination of cross-entropy loss and the KL loss.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Line 65: typo in Infact --> In fact\n- Line 252: typo in fianl --> final"}, "extra_text": {}}, {"review_id": "45K55GZTVE", "reviewer": "Reviewer_4H7e", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful comments. We answered your comments as per below.\n\n1.\tIt does not explain why supervised contrastive learning is used.\n\n[Answer] We have pointed out the motivation of using the supervised contrastive learning in both the line 65-68 of Page 1 and the line 302-307 of Page 6 in the previous manuscript. We aim to underscore how this approach aids in enhancing the discriminative power of our model by ensuring that it effectively captures the relationships between different modalities while respecting their class labels. To enhance clarity and address any potential oversight, we will revisit these sections in our manuscript and ensure that the explanation of why we utilize supervised contrastive learning is sufficiently detailed and prominent.\n\n2.\tThe most recent multi-modal rumor detection model compared is from 2020.\n\n[Answer] In our manuscript, we have incorporated baseline models EBGCN (Line 766) and MFAN (Line 773), which were published in 2021 and 2022, respectively. MFAN is recognized as a state-of-the-art model in this domain. These models represent the latest advancements in multi-modal rumor detection and serve as essential benchmarks for evaluating the effectiveness of our proposed approach. \nFurthermore, we have also conducted experiments involving ChatGPT. However, it's important to note that ChatGPT, while proficient in various natural language understanding tasks, is not optimized for the specific task of rumor detection. As we have verified, ChatGPT may not perform as effectively in debunking rumors compared to specialized rumor classification utilities like the models included in our baseline.\n   \n\n3.\tThe effectiveness of the student model has not been compared with other models, making it difficult to prove the effectiveness of distillation.\n\n[Answer] In our manuscript, we conducted experiments to evaluate the effectiveness of knowledge distillation (KD) on the student models. In Table 3 on Page 8, we have reported the performance of the student models before and after the application of knowledge distillation. Specifically, \"Student-2 initial\" represents the incomplete SOTA MFAN model with certain components removed, and \"Student-4 initial\" is another incomplete SOTA MFAN model with additional components removed. Following knowledge distillation, we compared the performance of these student models with the corresponding models before distillation. For example, \"Student-2 after KD\" and \"Student-4 after KD\" are the student models after knowledge distillation. These models were evaluated on both the Weibo and Pheme datasets. The results demonstrate that after applying knowledge distillation, the student models exhibit improved performance. For instance, \"Student-2 after KD\" achieved a 1.69% accuracy improvement on the Weibo dataset, and \"Student-4 after KD\" improved by 1.35%. Similarly, on the Pheme dataset, \"Student-2 after KD\" showed a 1.82% accuracy improvement, and \"Student-4 after KD\" improved by 1.56%.\n\nThese findings directly attest to the effectiveness of knowledge distillation in enhancing the performance of the student models. While we do not provide a comparison with other external models in this specific context, the improvements observed before and after distillation validate the positive impact of this technique on our student models.\n\n\n4.\tMain experiment focused on multi-modal scenarios; more experiments should be conducted starting from the perspective of missing modalities, beacuse the main motivation is incomplete modalities.\n\n[Answer] We have conducted more experiments as shown in Table 1 for the student model missing more modalities as follows:\nStudent-5: The model exclusively relies on the comment modality.\nStudent-6: The model exclusively relies on the visual modality.\nStudent-7: The model exclusively relies on the comment and visual modalities.\n\nTable 1: Performance comparison of knowledge distillation when facing more missing modalities.\n|                    | Weibo-2019 |           |        |          | Pheme    |           |        |          |\n|--------------------|------------|-----------|--------|----------|----------|-----------|--------|----------|\n|                    | Accuracy   | Precision | Recall | F1-Score | Accuracy | Precision | Recall | F1-Score |\n| Student-5 initial  | 69.49      | 70.51     | 64.63  | 64.46    | 74.29    | 70.63     | 60.74  | 60.40     |\n| Student-5 after KD | 72.20 (+2.71)      | 71.49 (+0.98)     | 69.49 (+4.86)  | 69.94 (+5.48)    | 76.36 (+2.07)    | 76.29 (+5.66）     | 62.32 (+1.58)  | 63.25 (+2.85)    |\n| Student-6 initial  | 71.18      | 71.78     | 67.01  | 67.27    | 73.77    | 80.34     | 55.83  | 52.97    |\n| Student-6 after KD | 73.22 (+2.04)      | 73.58 (+1.80)     | 69.66 (+2.65) | 70.19 (+2.92)    | 75.84 (+2.07)   | 80.39 (+0.05)    | 59.88 (+4.05)  | 59.63  (+6.66)  |\n| Student-7 initial  | 77.97      | 77.12     | 77.45  | 77.25    | 75.06    | 69.64     | 66.83  | 67.79    |\n| Student-7 after KD | 79.66 (+1.69)      | 78.99 (+1.87)     | 78.46 (+1.01)  | 78.69 (+1.44)   | 77.14 (+2.08)   | 72.76  (+3.12)   | 68.56 (+1.73)  | 69.89 (+2.10)   |\n\nIn addition, we illustrated model comparison between our model with the SOTA MFAN on two new larger multi-modal datasets, Weibo-2017 (Jin et al., 2017) and Twitter  (Boididou et al., 2018), as shown in Table 3 below. We will fill in the performance of other six baselines systems in the camera-ready version if the manuscript is accepted.\n\nTable 2: Statistics of the two new datasets. N: Non-rumors; R: Rumors.\n|            | #N   | #R   | #Images |\n|------------|------|------|---------|\n| Weibo-2017 | 4749 | 4779 | 9528    |\n| Twitter    | 6026 | 7898 | 514     |\n\nTable 3: Performance on the two new datasets.\n| Model           | Accuracy | Precision | Recall | F1-Score |\n|-----------------|----------|-----------|--------|----------|\n|       |          |    Weibo-2017       |        |          |\n| CLKD-IMRD(ours) | 92.94    | 92.99     | 93.03  | 92.94    |\n| MFAN            | 90.23    | 90.56     | 90.07  | 90.17    |\n|          |          |     Twitter      |        |          |\n| CLKD-IMRD(ours) | 91.09    | 92.14     | 88.95  | 90.15    |\n| MFAN            | 89.57    | 89.02     | 88.63  | 88.82    |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tProposes a multi-modal model as a teacher model to guide the single-modal model.\n2.\tThe experimental results are very comprehensive.\n", "reasons_to_reject": "1.\tIt does not explain why supervised contrastive learning is used.\n2.\tThe most recent multi-modal rumor detection model compared is from 2020.\n3.\tThe effectiveness of the student model has not been compared with other models, making it difficult to prove the effectiveness of distillation.\n4.\tMain experiment focused on multi-modal scenarios; more experiments should be conducted starting from the perspective of missing modalities, beacuse the main motivation is incomplete modalities.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a rumor detection model based on contrastive learning and knowledge distillation, which not only faces multi-modal scenarios but also addresses situations with missing modalities. In the teacher model, good classification results are obtained through the fusion of features between different modalities and supervised contrastive learning. By applying knowledge distillation, using the soft labels generated by the teacher model to supervise the student model, the effectiveness of the student model is enhanced in cases where modalities are missing.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of multi-modal rumor detection on social media, especially focusing on predicting with incomplete modality information. The method is based on constrastive learning for fusing the features across modalities and knowledge distillation from a teacher model trained for both modalities to improve performance when modalities are missing.

The reviewers appreciated that the paper introduced a new method for this task, especially dealing with the incomplete modality problem. Some of the reviewers also noticed that paper is well written and ablation experiments are provided.

All reviewers considered as one of the key negative points to use more than a single data set for each of the two languages, and ideally larger ones. The authors have provided additional results in the response on more data sets.
Decision: Accept-Findings