Beyond Detection: A Defend-and-Summarize Strategy for Robust and Interpretable Rumor Analysis on Social Media
Authors: Yi-Ting Chang
Yun-Zhu Song
Yi-Syuan Chen
Hong-Han Shuai
Venue: EMNLP 2023 Main (2023)
Keywords: rumor detection
social network analysis
adversarial attack
interpretability
summarization
unsupervised learning
self-supervised learning

Abstract: As the impact of social media gradually escalates, people are more likely to be exposed to indistinguishable fake news. Therefore, numerous studies have attempted to detect rumors on social media by analyzing the textual content and propagation paths. However, fewer works on rumor detection tasks consider the malicious attacks commonly observed at response level. Moreover, existing detection models have poor interpretability. To address these issues, we propose a novel framework named **D**efend-**A**nd-**S**ummarize (DAS) based on the concept that responses sharing similar opinions should exhibit similar features. Specifically, DAS filters out the attack responses and summarizes the responsive posts of each conversation thread in both extractive and abstractive ways to provide multi-perspective prediction explanations. Furthermore, we enhance our detection architecture with the transformer and Bi-directional Graph Convolutional Networks. Experiments on three public datasets, *i.e.*, RumorEval2019, Twitter15, and Twitter16, demonstrate that our DAS defends against malicious attacks and provides prediction explanations, and the proposed detection model achieves state-of-the-art.

Beyond Detection: A Defend-and-Summarize Strategy for Robust and Interpretable Rumor Analysis on Social Media
Yi-Ting Chang and Yun-Zhu Song and Yi-Syuan Chen and Hong- Shuai National Yang Ming Chiao Tung University, Taiwan yzsong.
(joshchang0111eeel yzsong ee07, yschen. ee09, hhshuai (nycu. edu. tw
Abstract
Critical Responses (18.9%)
As the impact of social media gradually esca- lates, people more likely to be exposed to indistinguishable fake news. Therefore, numer- ous studies have attempted detect rumors social media by analyzing the textual content and propagation paths. However, fewer works rumor detection tasks consider the malicious attacks commonly observed at response level. Moreover, existing detection models have poor interpretability. To address these issues, pro- pose a novel framework named Defend-And Summarize (DAS) based on the concept that responses sharing similar opinions should ex- hibit similar features. Specifically, DAS filters out the attack responses and summarizes the responsive posts of each conversation thread both extractive and abstractive ways to pro- vide multi perspective prediction explanations. Furthermore, we enhance our detection archi- tecture with the transformer and Bi-directional Graph Convolutional Networks. Experiments on three public datasets, i.e., RumorEval2019, Twitter15. 15, and 1witter16, demonstrate that our DAS defends against malicious attacks and pro- vides prediction explanations, and the proposed detection model achieves state-of-the-an
Prodicted Probabillity
falie
Enue
Fied
Number of Responses
Figure I: Three examples for the predicted probabil- ity of each class with respect to the responses the Twitter15 dataset. The curves with their face colored represent the ground-truth labels for their source post. Critical responses that result in prediction shifts larger than 0.5 are marked with a circle.
corresponding responses (Ma et al., 2016; Volkova et al., 2017). In addition, several studies show the importance of considering the propagation path be- tween the responses within the same conversation thread (Ma et al., 2017, 2018; Lu and Li, 2020). To better extract information from the propagation, Graph Convolutional Networks (GCNs) are widely adopted and achieve remarkable performance for the rumor detection problem (Bian et al., 2020; Wei et al., 2021; Sun et al., 2022). For instance, Song et al. (2021) pioneer the integration of transformer and GCN to better detect the rumors.
1
Introduction
Due the low cost and easy access information, social media has become popular platform for in- formation dissemination. However, it increases the spread of misinformation as well (Vosoughi et al., 2018). The spread of rumors could cause panic and further damage public mental health or lead to se- vere economic loss (Verma et al., 2022). Therefore, debunking unverified rumors the Internet has become indispensable issue (Ahsan et al., 2019). Numerous researchers have been dedicated de- tecting rumors automatically. Early works mostly rely on the textual content each post and the
However, two main challenges remain unad- dressed. Firstly, the detectors could be sensitive critical responses toward an event, i.e., responses that significantly impact the detectors. Fig. demonstrates that roughly 18.9% of posts in the Twitter15 dataset contain critical responses. The influence of such responses may be formulated as attacking manner by adversaries. As prior stud- ies mainly focus on determining the veracity of given claim by the source post and responsive posts, the potential threat from the attack responses could lead to vulnerability in detection models (Le et al., 2020). Hence, some works have developed GAN-
'The source code and our datasets are available at hitps: / /ggthth.ccm/////c. goggthuhccom/joschhngggll. 1/EMNLP2023, 22023-RmmorDDA RumorDAS.

---

style frameworks to build more robust detectors (Ma et al., 2019, 2021; Song et al., 2021). How- ever, retraining the entire model to defend against attacks could be time-consuming and limited to rec- ognizing only adversarial examples, disregarding various forms of real-world malicious attacks.
interpret model predictions from different perspec- tives. Representative responses are then extracted from the medoid of each cluster. Afterward, the response abstractor aims to produce more compre- hensive and human-understandaall explanations by summarizing the responses from cach cluster. We exploit the pre-trained abstractive summariz- and transfer the models to the rumor detection corpora via self-supervised learning. In particular, the abstractor is finetuned by cluster-summary pairs where the medoid of each response cluster serves as a pscudo summary. Combining extractive and ab- stractive summaries from DAS provides detectors and users with more reliable and comprehensive information on different viewpoints. Moreover, we introduce Bi-directional Gransformer-Graph Network (BiTGN) improve rumor detection by integrating the robust textual representations of the transformer and the structural information of Bi- directional GCN (BiGCN). The contributions of this paper are summarized as follows:
On the other hand, recent works mainly lever- age neural networks for predictions, making the interpretability behind those predictions unattain- able due to the black-bor box property of such models (Ghorbani et al., 2019). To better interpret the detectors' behavior, some works utilize attention mechanism to highlight the important parts of the inputs (Khoo et al., 2020; Lu and Li, 2020), which demonstrates the feasibility of probing the detec- tion models by identifying influential responses in a conversation thread. However, such an approach lacks comprehensive and human- uumaan--uddestaddbble clues, which brings the second challenge of pro- viding organized explanations that cover different viewpoints. We posit that the consideration of multiple perspectives within the discussion threads serves to enhance readers' cognizance of a multi- tude of viewpoints, thereby discouraging the uncrit- ical acceptance of an excessively confident verdict.
We propose novel framework named DAS that reduces the model vulnerability and provides pre- diction explanations without additional annota- tions and retraining of the detection models.
In this paper, we propose a novel framework called BrrenddAndSSummaane detector vulnerability and provide prediction ex- planations. The design of DAS follows the idea that responses with similar stances viewpoints should lie closer in the embedding space. This concept substantiated by prior studies (Darwish, 2019; Rashed et al., 2021), which showcase that various standpoints of political opinions on Twitter can be well partitioned into distinct clusters based on the embedding representations. This character- istic could enhance summarization with more struc- tured and comprehensive information. As such, DAS includes a response extractor and a response abstractor. The extractor filters and organizes the responses, while the abstractor condenses the in- formation according to the organized responses. To improve robustness, we preemptively mitigate malicious attacks with the response extractor. By exploiting the idea of anomaly detection, we filter the responses by considering the genuine ones as normal data and the attack ones anomalies. In ad- dition to removing the potentially risky responses, we further organize the remaining responses to find representative ones. We apply clustering to auto- matically explore the underlying aspects of data to
We explain model predictions with extractive and abstractive summaries by incorporating the con- cept of clustering into self-supervised learning.
(DAS) to reduce
Experiments on three public datasets show that DAS defends against attacks while producing multi-perspectivv explanations, and the proposed BiTGN achieves state-of-the-ee rumor detection. Human evaluation further demonstrates the inter- pretability of the generated summaries.
2 Related Work
Model Vulnerability Adversarial attack has been used to simulate the impact of critical responses (Xu et al., 2021: Mehrabi et al., 2022: Xie et al., 2022). For example, Ma et al. (2019, 2021) develop GAN-style frameworks to generate conflicting ut- terances that complicate the original conversation threads and force the discriminator to capture more robust text features. Moreover, Le et al. (2020) model such influential responses scenario and propose an adversarial comment gen- erator to mislead the detector. To reduce the model vulnerability to such attacks, Song et al. (2021) perform adversarial training on the detector with
novel attack

---

Overall Framework
Oetecor
Rumor Detector (BITGN)
Detector Head
BART
Extractive Summary
Suppaiqu undieess
00 Siit di Auppert
Rumer Octector (Emzsder)
Response Generator
Transformer Encoder
(Deuxder)
Summary
Defend-And
Refend-AAnd-Summariiu Summarize (DAS) Framework
Filter by reconstruction error
representative resporses
Response Extractor
Reconstructin Error
k-masns Clustering
Responde
Extractive Summary
Encoder
Decode
Abstractor
Abstractive Summary
Figure 2: Overview our proposed framework (upper left). The rumor detector BiTGN (upper right) trained predict the veracity of each source post. The response summarizer DAS (lower) preemptively filters out attack responses generated by the response generator. It then organizes the remaining responses into k clusters and produces both extractive and abstractive summaries for each cluster accordingly. During the inference phase, the detector makes predictions based the source post and the summaries.
adversarial responses. However, this approach re- quires retraining the entire model. contrast, our work presents a novel framework that resists re- sponse attacks without retraining the model.
et al., 2008; Potthast et al., 2018) or hand-crafted features (Yang et al., 2012: Liu et al., 2015; Ma et al., 2015: Wu et al., 2015). In recent years, deep neural networks such as CNN and RNN have been widely adopted to extract the text features auto- matically (Volkova al., 2017; Ma al., 2016). Furthermore, Recursive Neural Networks (RvNN) (Ma et al., 2018) and GCN-based upproaches (Bian et al., 2020; Wei et al., 2021; Sun et al., 2022) are proposed to analyze the propagation structures of rumors. In addition, recent line of studies also leverages Transformers (Khoo et al., 2020; Song et al., 2021; Tian et al., 2022) to capture long- distance interactions between responsive posts.
Interpretability One class of studies typically explains model predictions by analyzing the atten- tion given to different parts of inputs, which is usually accomplished by visualizing the word im- portance scores (Ribeiro et al., 2016; Vig, 2019) or using heatmap (Samek et al., 2017). For instance, Lu and Li (2020) visualize the attention weights between source tweets and the propagation struc- tures to highlight evidential words and suspicious users in predicting fake news. Similarly, Khoo al. (2020) provide token-level and post-level ex- planations by examining the attention weights of transformer layers. Apart from attention based ap- proaches, Pugoy and Kao (2021) explain recom- mender system predictions by producing extractive summaries from user and item reviews, which cap- ture crucial sentences for models and provide more comprehensive information than word-level and review-level explanations. Consequently, in this paper, attempt provide realistic explanations for rumor detection models by summarizing differ- ent opinions each conversation thread.
Methodology
3.1 Problem Formulation
Here, we first define the notations. A conversation thread denoted by X {>:} where zg is the source post containing the main event to veri- fied, and {x, ) represents the responsive posts of on- A graph G = (V, E)) with vertex set V and edge set E is formed by taking each post in X as a node, and the responsive relations between nodes further define the edges. Specifically, two nodes x; and æú are connected by an edge eij E if one of them responds the other one. The ground-truth label denoted by y E Y = (N, F,T, U} (i.e., Non-Re Non-Rumor, False Rumor, True Rumor, and Unver- ified Rumor). In our framework, the rumor detector
Rumor Detection Early studies tend to verify the truthfulness of social media posts based on either traditional language processing skills (Badaskar

---

aims predict the veracity of the source post æo with or without attacks, while the response summa- rizer aims to extract k representative responses and produce abstractive summaries accordingly.
and BU-GCN are A A and A A A re- spectively. The feature matrix is iteratively updated by each GCN layer both directions. As such, the aggregated feature 7TD from the TD-GCN with Iig layers is obtained as follows,
3.2 Rumor Detector
7TD MEAN(ZE,)
We first introduce the proposed Bi-directional Transformer Graph Network (BiTGN) for ru- mor detection, which integrates the advantages of the transformer network and the Bi-directional Graph Convolutional Networks (BiGCN) as de- picted in Fig. 2. Previous studies have shown that Transformer-based models are more robust to out- of of-distribution data (Hendrycks et al., 2020) and adversarial attacks (Jin al., 2020) compared to conventional models such as CNN and RNNs. To obtain robust textual feature, we adopt trans- former encoder Bene with Le layers to encode all posts in a conversation thread by concatenating them as a sequence. Specifically, the post content is first transformed into vector representations by an embedding layer. Let h! E Rlaiixd xd denote the embedding of the i-th post 2; with dimension d.
z - oAA (
(3)
A - (A +IJD
where 2/60 Z, A normalized adjacency matrix with self- celf-connection, D; = Ej= A represents the degree of the i-th node, and W e Rdddd is a learnable matrix. Similarly, the aggre- gated result for BU-GCN is obtained by substi- tuting A ATD to A BU in Eq. (3). In the final step, the aggregated features ZTD and 2 are concate- nated and passed through fully connected layer and a softmax function as follows,
y = softmax ( LT BU W + +),
(4)
where W E 2ddx|1| and b E B are trainable pa-
The embedding of a conversation thread ( can be represented as follows,
rameters and ý is a vector indicating the predicted probability of each class.
H (0) (b h
(1)
3.3 Adversarial Response Generator
To simulate the attack responses from various users in real-world scenarios, we adopt an Adversarial Response Generator (ARG) proposed by Song et al. (2021). Given a conversation thread {x, {-, ARG generates an adversarial response =; that makes the detector deviate from the ground-truth y by maxi- mizing the detection loss Ldet, detailed in Sec. 3.5. Notably, ARG shares the encoder Bene with BiTGN and takes the hidden representation of the last en- coder layer (I.)) as inputs. Moreover, the gener- ated response " is then attached to the source post 2o to update the adjacency matrix, and its represen- tation h; is concatenated with the embedding of n-1 to serve part of the encoder's inputs." 2 {a; [=
where stands for the concatenation. Next, the em- beddings are iteratively fed into each encoder layer which consists of Multi-Head Attention (MHA) (Vaswani et al., 2017). The hidden representa- tion at the 1-th transformer layer denoted by ( MHA (H (I) Note that since we con- catenate all posts in a thread, different posts can attend to each other and exchange information dur- ing the encoding process. After the text encoding process, we obtain the node feature 2, by taking mean-pooling all the token representations of the i-th post from the last encoder layer. The hid- den feature matrix is obtained as follows.
Z 2 z,
|| ... [,,] E Innd
(2)
3,4 Defend-And--Smmariiz Framework
Defensive Response Extractor (DRE) To de- fend against the attacks, we aim filter out the attack responses simulated by ARG. We hypoth- esize that if one malicious response can mislead the detector, it must deviate from other normal re- sponses in the embedding space. Therefore, we adopt an autoencoder (AE) to detect the anomalies
To further aggregate the contextual features with the structure of responses, we leverage GCN- based model fyen to capture the interactions be- tween different posts in two directions, which consists of Tow-Down GCN (TD-GCN) and Bottom-Up GCN (BU-GCN). Let A E RR de- note the adjacency matrix where Aij = 1 if x re- sponds to Z;. The adjacency matrices for TD-GCN
MMore details of ARG are provided in the Appendix.

---

according the reconstruction error. Concretely, we initialize the encoder Pext-e and decoder Pext-d of the AE with transformer layers and train the model on normal responses. The reconstruction process of a response z; is as follows,
responses can be approximated the summary of the whole thread. However, such kind of set- tings suffers from a great portion of inappropri- ate summary pairs when the responses cover various aspects. As a result, we create pseudo summaries from the cluster results obtained from DRE. Specifically, each medoid h is taken the pscudo summary, and the remaining responses of cluster Cj are concatenated the inputs for producing the summary, i.e.,
fert-f1 (Deat (h,)),
(5)
h, dezt-a (eent- (ened((een)((z))) f2(z)),
where Pext-f1 f1 and fe represent fully connected layers. E [l is the hidden noise with di- mension d, <
abs = Rabs [hill-- by il byyn In. (hll)) (7) 5
compress the features. We apply Le loss to calculate the reconstruction er- ror and select the top-m responses with the least loss since an unseen attack response should cause a more significant error. The selection number is determined by pre-defined extract ratio ft [p x n]. After the filtering process, the remaining re- sponses [ to obtain the response represen- tations {ri})=}- Afterward, k-means clustering is performed on these representations to capture the intrinsic perspectives of different responses. The responses are partitioned into k clusters by mini- mizing the intra-cluster sum of distances from each sample to its nearest centroid. Let {C,}/=1 denote the set of & clusters. The extractive summary is formed by combining the medoids of all clusters, where a medoid reat the response closest cluster's centroid Cy:
where h is the summary embedding of cluster
Cj. Habs = [h ||... || hg represents the em- bedding of the abstractive summary. Note that all responses in a cluster are treated as inputs during inference. Besides, each abstractive summary is attached to the source post 2o maintain the tree structure, where the new adjacency denoted A'. Finally, both extractive and abstractive results input along with zo for rumor detection:
we take the mean-pooling
m
y = ggen (eenooeeeeln Hext
Habel), A'). (8)
3.5 Training Objectives
We train the rumor detector and adversarial re- sponse generator in two stages. The trainable pa- rameters for the detector are the encoder layers fenc and the GCN layers 0 gen. while the decoder layers Odcc are only trained for ARG. In the first stage, the generator is trained with the detector improve the detection results and its generation quality. The objectives of the generator are the cross entropy for rumor classification LCE and the cross entropy for text generation List Σ jectives of the first stage calculated as follows,
argmin((lr: -ailrr--;l1)). MEC
(6)
Finally, the embedding of the extraction result is de- noted by Hent - [hart] I... ||. h ||...|| hg where hert is the response embedding the medoid pont Note that some responses may lose their parent node after the extraction. Thus, we assign a new parent node for such response by recursively track- ing back until finding remaining node.
-- 1 2; log क.. The ob-
Lace Oyen) = Lon(F,y), Legen (fdcc) = LCE(S.y) + Tn); List = Ladet + Lgen-
(9)
Self-Superviee Response Abstractor (SSRA) One main challenge of training the response ab- stractor is the lack of ground-truth summary labels. Inspired by previous works (Wang and Wan, 2021; Elsahar et al., 2021), we finetune our SSRA fans un- der self-supervised settings. Previous works often utilize Leave-One-One (LOO) settings where each response in conversation thread takes turns to be the pseudo summary. This approach follows the assumption that responses of the same thread focus on the same event, and each of the high-relevance
In the second stage. we train the generator with fixed detector. The target of the generator is to pro- duce adversarial response that degrades detector's performance while resembling human writing style. Thus, is optimized to maximize the cross entropy for rumor detection while minimizing Lint-
Lznd Lgen(Osec) =-CoE(F,v) + Lial (Zng
(10)

---

RE:0019
Twitter15
Twitter16
Summarizer
Atfuck
Acc.
m1
ASR 4
Diff. ASR
Acc.
mFL
ASR
DSf
Acc.
m.Fi
ASR
Diff
64.57
64.29
85.64
85.50
84.47
84.40
19.76
62.32
27.40
DRE (p=0.25, k=3)
56.51
55.82
22.35
75.62
71.76
72.37
-52.61
DRE 15,0.15, km3)
V
58.07
57
21 66
-40.66
78.69
79.02
11.97
-51.91
75.67
76.04
15.32
-56.87
DRE (p=0.05, k=3)
39.63
58.95
19.24
-13.08
80,07
80.26
10.67
-53.1
77,02
77.25
13.74
-8845
DAS (p=0,25, k=3))
57.85
57.08
21.96
-40.36
72.78
73.37
18.71
-5.17
70.16
70.44
21.72
-5047
(p-0.15, k=3)
17.93
76.53
DAS (p-0.05, kn3)
60.96
59.98
17.96
-44.36
77.32
77.39
18:24
-50.14
74,94
75.07
15.82
-56.37
Table 1: Overall results of adversarial attack defense on BiTGN. Diff represents the difference of ASR with and without defense. Both DRE and DAS can successfully resist a large amount of attacks from ARG.
For the DAS framework, the trainable parameters are the response extractor dext and the response abstractor Pabs- The response extractor is trained to reconstruct the embedding of normal responses by minimizing the L2 loss between the original embedding h; and the reconstructed one hą:
Macro
ASR
DAS, k-2
066
005
DAS, k-4
0.6
0.4
wio DAS, w/o attack
w/
0.2
0.3
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.1
(ent() = C2 (h,,hh)
(11)
Figure 3: Effect of extract ratio p and number of clusters k Twitter1.. The dashed lines represent the detec- tion performance without DAS. The Macro-F1 (left) increases the extract ratio p decreases and the Attack Success Rate (right) behaves in an opposite trend. The number of clusters & does not influence the results sig- nificantly, which demonstrates the robustness of DAS.
The response abstractor is optimized to minimize the cross entropy between the generated summary o and the pseudo summary sget i.e.
Labo (Outt))))) = Ciea(s (gge, ggat).
(12)
4 Experimental Results
4.1 Experimental Setup
conversation thread using tree-structured recursive neural networks with GRU units. BiGCN (Bian et al., 2020) represents each post with TF-IDF vec- tors and utilizes a bi-directional GCN to aggre- gate both propagation and dispersion structures. EBGCN (Wei et al., 2021) extends the BiGCN by adaptively updating the weight of cach edge formed by the responsive relations with Bayesian approach. WETGN (Song al., 2021) combines the transformer encoder with a unidirectional GCN and adopts an edge filter. DUCK- -UT (Tian et al., 2022) models each conversation thread as a graph, a stream, and a user tree (UT) separately. Due to the lack of such user information, we climinate the user tree (-UT) for a fair comparison. Note that we run these baselines our version of Twitter 15 and 1witter16 datasets since the number of responses not comparable with previous works.3
Datasets We evaluate our model on three real- world public datasets. Twitter15 and Twitter16 datasets (Ma et al., 2017) respectively contain 1490 and 818 Twitter posts labeled with Non-Ruman (N),
True Rumor (T), False Rumor (FF), and Unverified Rumor (U). Moreover, RumorEva12019 (RE22119) dataset (Gorrell et al., 2019) was released by the SemEval workshop in 2019, which contains 446 posts from both Twitter and Reddit. It provides three veracity labels, i.e., True Rumor (T), False Rumor (F) and Unverified Rumor (U). The detailed statistics of datasets are listed in the Appendix.
Evaluation Metries For the generation quality of the response abstractor, we calculate the perplexity (PPL) by GPT-2 (Radford et al., 2019) and the factual consistency by FactCC (Kryscinski et al., 2020). For rumor detection, we report the accuracy (Aec.) over all classes, the F1 score of each class and the macro-averaged Fi (mFi).
Baselines We compare the performance of rumor detection with several baselines. RyNN (Ma et al., 2018) captures the propagation patterns of each
SSinee Twitter15 and Twitter16 only provide the tweet IDs for responses, we use the version released by Song et al. (2021), where the textual content of cach response is manually obtained via Twitter API.

---

RR2019
Twitter15
Twitter16
Part A (Inoormativeness)
Summarizer
P
FaciCC
PMI.
FactCC
PPL.
FactCC
Median
BART-base- SAMSS
0.63
68.77
0.87
62.40
0.55
59.18
SSRA-LDO
Mean
SSRA-LO0
4.11
71.02
4.20
71.27
7:20
73.72
55RA-k-means
SSRA- AeRA-ssmams (t=1)
2.50
72.11
3.39
78.15
3.03
77.89
SSRA- (l=2)
82.29
2.55
82.97
SSR A-mmeas (lm3)
2.32
83.01
2.33
87.64
2.44
82.93
Part B Predictions per sample)
Correct
Wrong
SSRA-I-means (1=4)
2.23
83.66
75.2%
Resporse
24 8%
S5R (t=5)
2,70
85.74
2.14
86.85
2.39
83.03
Summary
10%
50%
80%
50%
100%
Table Automatic evaluation of generated summaries, The best / second best scores are marked in bold / underlined. Our SSRA-k-means models generate sum- maries with better text quality and factual consistency.
Figure 4: Human evaluation of generated summaries. In part A, our SSRA-k-mean model can generate more in- formative response summaries compared SSRA-L.OO In part B, human predictions based on either responses or summaries can achieve comparable accuracy, which demonstrates the interpretability of the summaries.
4.2 Adversarial Attack and Defense
We first discuss how the proposed DAS framework can reduce detector vulnerability. Table 1 demon- strates the results of BiTGN using BART encoder, and under attuck by ARG while equipping differ- ent response summarizers. Apart from the accu- racy and mF1, we also calculate the Attack Suc- cess Rate (ASR) of ARG, defined as the ratio of successfully misled predictions among all initially correct predictions. The first and second rows rep- resent the performance before and after the attacks. The results manifest that ARG indeed degrades the detection performance since the attack success rates are greater than 60% on all datascts. Next, we compare the defensive ability of our response summarizers (both DRE and DAS) with different extract ratios P given the number of clusters ie set to 3. First, the model performance has been recov- ered significantly by simply equipping DRE during inference, indicating that the extractor can filter out a large portion of the attack responses generated by ARG without retraining. Similarly, DAS can also defend against the attacks while additionally providing prediction explanations with abstractive response summaries. To further analyze the behav- ior of DAS, Fig. shows the model robustness with different extract ratio p and number of clusters on Twitter15. 15. First of all, the Macro-Fi increases with the extract ratio decreases and saturates when the extract ratio p is around 0.2. Besides, even with a high extract ratio, i.e., the left part of both figures, the model could still defend against a certain ratio of attacks while preserving more information from the responses. Secondly, increasing the number of clusters to produce more diverse summarics only slightly affects the defense ability of DAS, which demonstrates the robustness of our model.
4.3 Interpret Predictions with Summary
Automatic Evaluation Here, we show that the generated response summaries can be used to ex- plain model predictions. Since higher text quality can help humans understand models' decisions, we first compare the generation quality of the ab- stractors trained under different sule-supe settings, including Leave- means with varying values of k. We initialize all models with BART-base-SAMSSI
(LOO) and k-
a summarizer pre-trained on the SAMSum corpus, 5 and take its zero-shot results as a baseline of text quality. Re- sults in Table 2 manifest that self-supervised mod- els demonstrate higher perplexity than BART-base- SAMSum due to the abbreviations and informal expressions in social microblog text, such as hash- tags and URLs. Compared SSRA-LOO, our pro- posed SSRA-k scores, indicating its ability to generate more fluent and human- understandable summaries. The inclu- sion of fragmented responses and incomplete sen- tences in SSRA-LOO training targets contributes to a higher perplexity. Secondly, we validate the factual consistency between the input responses and generated summaries. We observe that SSRA- k-means outperforms baselines across different h values, suggesting the necessity of self-supervised learning. Besides, even when k = i.e., SSRA-K- means only provides one summary as SSRA-LOO does, our model scores higher than SSRA-LOO on all datasets by covering more factual information from the responses. Furthermore, the factuality im- proves significantly when & > 1, demonstrating the effectiveness of providing the abstractor with
achieves better perplexity
'Due to the space constraint, the results of Twitter16 and RE2019 are provided in the Appendix.
"The SAMSum dataset (Gliwa et al., 2019) contains about 16K daily conversations with ground-truth summary labels.

---

Source Post (False Rumor): leaked phone call between rebel leader & russian intel agens: "cossicks" shot down 4MH17. URL URL
RE2019
Twitter15
Twitter16
Model
Acc.
mFi
Responses ... [5]: @namel @name2 who leaked this? Do you check sourves? Have verified anything? ... [7]: RT G'mmmll: NEW. Leaked phone call between rebel leader Russian intel agens: "Cussacks" shot down RMH17 URL [15]: @namel @name3 NEW. Leaked phone call between rehel Russian intel agens: "Cossacks" shot
RvNN
52.00
51.42
73.88
73.81
75.18
75.22
BIGCN
60.31
56.83
84.02
$3.69
87.33
87.07
ERGON
57.45
52.91
82. 79
85.43
85.16
#MH17 URL.
WETGN
68.82
67.53
87.35
$2.34
87.16
87.09
[19]: @namel @ d'namoe what is the proef [20]: @ nnamel Jin, Jim, Jim, of
DUCK
69.73
69:02
86.25
86.14
86.55
86,40
you promoted Opps there goes our credibility again Too bad @rname5
on you,
BITGN
70.84
70.05
87.77
87.73
89.10
89.02
BUGCN
69.27
87.72
88.02
Summary [1]: RE NEW Lecked phone call betweeen rebel leader & Russimn intel agent: "Cossacles" shot down #MH12 [2): This hoax, hoax of sorts and you promoted it. Shame on you [3]: what is the prout??
TDGCN
69.50
69:03
NK. 39)
80.35
87.78
87.79
GCN
70.40
69.48
87.35
87.42
87.89
87.83
Table 4: Overall results of rumor detection. The best / second best scores are marked in bold / underlined. Our BiTGN outperforms baselines in the first block.
Table 3: Generation example of SSRA-k-meatt (k=3), Key information captured by summaries is highlighted.
responses from different perspectives.
between Pyt of responses and summaries. We ob- serve Pearson correlation of 0.54 with p-value 0.014, justifying a high correlation between pre- dictions based on responses and summaries. This demonstrates that the response summaries effec- tively capture crucial information and can interpret human decisions based on the responses. Table 3 demonstrates generation example of SSRA-K- means (k- = 3). The source post contains a false claim about "Malaysian flight shot down by Cos- sacks". Our summaries encompass diverse stances such as "It was hoax" (deny) and "what is the proof" (query), providing evidential guidance for models and users to evaluate the veracity of the source post. These summaries also help identify which information models focus on.
Human Evaluation and Case Study We recruit 100 human readers and conduct two parts of user study, In part A, we randomly select 10 samples from each of the three datasets, each containing a source post, responses, and two sets of sum- maries generated by SSRA-LOO* and SSRA-k- means (k- 3). Readers are requested to assess the informativeness of the summaries based on the viewpoint coverage and diversity using a Likert scale from to 5, with 5 representing the most in- formative. In part B, we aim to evaluate whether humans make a consistent judgment, either after reading responses or response summaries. Thus, we select 20 samples, including 10 truc and false rumors, and ask the participants to judge the truth- fulness of source posts based the provided in- formation, Le.,, responses or summaries. The upper of Fig. 4 shows that SSRA-k --eans outperforms SSRA-LOO in terms of informativeness, indicat- ing the effectiveness of utilizing k-means cluster- ing to grasp diverse opinions. The results of part B show that participants who solety read the sum- maries achieve comparable accuracy to those who read the responses average, with marginal difference of approximately 5%. Although the re- sponses provide more complete information, these findings suggest that summaries are practical for social media users to judge the post veracity as the summaries effectively capture different view- points in a shorter format. In real applications, we could provide both responses and summaries to convey the essential viewpoints by summaries and delve into details in the responses. Moreover, we evaluate the percentage of ground-truth predictions Pgt for each sample and analyze the correlation
4.4 Rumor Detection
In this section, we purely analyze the rumor detec- tion results of the proposed BiTGN with RoBERTa encoder in Table 4.7 Compared with all base- lines, our BiTGN achieves the best accuracy and macro-averaged F1 on all datasets. Specifically, transformer- models (BiTGN, DUCK-UT. WETGN) outperform models that use TF-IDF vec- tors (BiGCN, EBGCN, RvNN) as node features, which demonstrates the importance of robust tex- tual representations. Moreover, our model strikes the best among wansformer-based baselines, show- ing that the BiGCN component can better aggregate the conversation information from two directions. Besides, compared DUCK -UT that models the conversation structures with two branches of trans- former networks, our model still achieves better results with fewer parameters. We also analyze the influence of GCN by ablating the BiGCN as shown in Table 4. In particular, the model with BiGCN
"We randomly split the responses into 3 groups and make SSRA-LOO generate one summary for each group.
TThe detailed statistics are shown in the Appendix.

---

(BiTGN) achieves the best performance on RE2019 and T while achieving the second best on 1witter15. We notice that the model without top- down GCN (-TDGCN) improves on the classes of non-rumor and true rumor of 1witter15. This may be caused by the diverse structure of these data, observed by Huang et al. (2020). Although the structure information may be noisy, the model can still benefit from introducing the information of propagation path through GCN layers compared with the model without GCN layers (-GCN).
this possibility as a direction for our future work.
Ethies Statement
We discuss some potential risks that our rumor de- tection system might raise. As our proposed frame- work highly relies on the interactions between dif- ferent users on social media, the content of the users' utterances including mentions to other users will be revealed to the system. However, the system doesn't I require any personal information such as user description, user account age, number of fol- lowers, number of posts, etc. Due to this reason, the proposed method shall not infringe on individual's privacy. Another risk is that the detector might still give wrong classification results that mislead the users. For this issue, we believe that our method could provide a more simplified but comprehensive summary of the diverse responses under each post, making the users able to observe the opinions on more sides toward an event. This might enhance the ability of the public to rethink and justify the truthfulness of various sources of information.
5 Conclusion
this paper, we propose a novel response sum- marization framework, Arfend-And-Summaald (DAS), to enhance the robustness and interpretabil- ity of rumor detection models. Our DAS frame- work is built around two pivotal components: De- fensive Response Extractor (DRE) adept at sifting malicious responses and extracting significant ones, and Super-upe Response Abstractor (SSRA) capable of producing multi-perspective ab- stractive response summaries. Moreover, we pro- pose a Bi-dirrctional Transformer Graph Network (BiTGN) to strengthen transformer-based detec- tors with bi-directional graph aggregation. Exper- iments on three real-world datasets demonstrate the potent capabilities of DAS to improve the re- silience and interpretability of detection models. Besides, BiTGN delivers state-of-the-art rumor de- tection. The combination of DAS and BiTGN sig- nals promising advancement in the field of rumor detection, providing a robust and interpretable solu- tion primed for future challenges and applications.
A Aknnowledgements
This work was supported in part by the National Science and Technology Council of Taiwan un- der Grants 1STCC-12-2221---449-000 NSTC-111-2221 -22-1-1---499164
and
References
Mohammad Ahsan, Madhu Kumari, and TP Sharma. 2019. Rumors detection, verification and controlling mechanisms online social networks: A survey. Online Social Networks and Media, 14:100050.
Limitations
Sameer Badaskar, Sachin Agarwal, and Shilpa Arora. 2008. Identifying real or fake articles: Towards bet- ter language modeling. In Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-11.
Our work focuses on determining the truthfulness of source post from social media websites by an- alyzing the structural information of its responses. Since the opinions from various users provide rich information and can influence other readers sig- nificantly, we do not consider the settings of fake news detection that solely rely on the news content. Moreover, the DRE component in our framework adopts the widely-used k-means algorithm to pro- duce response clusters without considering specific aspects. However, it would be beneficial to create clusters with more fine-grained aspects, such as the stance or sentiment of responses. This would enable humans to gain a more comprehensive un- derstanding of the public's opinion. We'll explore
Tian Bian, Xi Xiao, Tingyang Xu, Peilin Zhao, Wen- bing Huang. Yu Rong, and Junzhou Huang. 2020. Rumor detection on social media with bi-directional graph convolutional networks. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 549-556.
Shuyang Cao and Lu Wang. 2021. CLIFF: Contrastive learning for improving faithfulness and factuality in abstractive summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6633-6649, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

---

Xiuying Chen, Hind Alamro, Mingzhe Li, Shen Gao, Xi- angliang Zhang, Dongyan Zhao, and Rui Yan. 2021. Capturing relations between scientific papers: An abstractive model for related work section generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 Ith International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 6068-6077, Online. Association for Computational Linguisties.
Ling Min Screna Khoo, Hai Leong Chieu, Zhong Qian, and Jing Jiang. 2020. Interpretable rumor detection in microblogs by attending user interactions. In Proceedings of the AAAI conference on artificial in- telligence, volume 34, pages 8783-8790.
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332-9346, Online, Association for Computa- tional I. Linguisties.
Kareem Darwish. 2019. Quantifying polarization on twitter: the kavanaugh nomination. In Social Infor- matics: 11th International Conference, SocInfo 2019. Doha, Qatar, November 18-21, 2019, Proceedings pages 188 208- Springer.
Richard Landis and Gary Koch. 1977. The mea- surement of observer agreement for categorical data. biometrics, pages 159-174
Hady Elsahar, Maximin Coavoux, Jos Rozen, and Matthias Gallé. 2021. Self-supervised and controlled multi-document opinion summarization. In Proceed- ings of the 16th Conference of the European Chap- ter of the Association for Computational Linguisties: Main Volume, pages 1646-1662, Online. Association for Computational Linguistics.
Thai Le, Suhang Wang, and Dongwon Lee. 2020. Mal- com: Generating malicious comments to attack neu- ral fake news detection models. In 2020 IEEE Inter- national Conference Data Mining (ICDM), pages 282-291. IEEE.
Xiaomo Liu, Armineh Nourbakhsh, Quanzhi Li, Rui Fang, and Sameena Shah. 2015. Real-time rumor debunking on twitter. In Proceedings of the 24th ACM international on conference information and knowledge management, pages 1867-1870.
Amirata Ghorbani, Abubakar Abid, and James Zou. 2019. Interpretation of neural networks is fragile. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3681-3688.
Bogdan Gliwa, Iwnna Mochol, Maciej Biesek, and Alek- sander Wawer: 2019. SAMSum corpus: A human- annotated dialogue dataset for abstractive summa- rization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70-79, Hong Kong, China. Association for Computational Linguis- tics.
Yi-Ju Lu and Cheng-Te Li. 2020. GCAN: Graph-aware co-attention networks for explainable fake news de- tection on social media. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 305-514, Online. Association for Computational Linguistics.
Jing Ma. Wei Gao, Prasenjit Mitra, Sejeong Kwon, Bernard J. Jansen, Kam-Fai Wong, and Meeyoung Cha. 2016. Detecting rumors from microblogs with recurrent neural networks. In Proceedings of the Twenty-Fiit International Joint Conference on Arti- ficial Intelligence, IJCAI' 16, page 3818-3824.
Genevieve Gorrell, Elena Kochkina, Maria Liakata, Ah- met Aker, Arkaitz Zubiaga, Kalina Bontcheva, and Leon Derezynski. 2019. SemEval-2019 task 7: Ru- mourEval, determining rumour veracity and support for rumours. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 845 854, Minneapolis, Minnesota, USA. Association for Com- putational Linguistics.
Jing Ma, Wei Gao, Zhongyu Wei, Yueming Lu, and Kam-Fai Wong. 2015. Detect rumors using time se- ries of social context information microblogging websites. In Proceedings of the 24th ACM interna- tional on conference on information and knowledge management, pages 1751-1754.
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. 2020. Pretrained transformers improve diitr--diitribution robustness. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 2744-2751, Online. Association for Computa- tional Linguistics.
Jing Ma. Wei Gao, and Kam-Fat Wong. 2017. Detect rumors in microblog posts using propagation struc- ture via kernel learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 708-717, Vancouver, Canada. Association for Computational Linguistics.
Qi Huang, Chuan Zhou, Jia Wu, Luchen Liu, and Bin Wang. 2020. Deep spatial- temporal structure learn- ing for rumor detection on twitter. Neural Computing and Applications, pages 1-11.
Jing Ma. Wei Gao, and Kam-Fai Wong, 2018. Rumor detection Twitter with tree-structured recursive neural networks. Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 1980-1989, Melbourne, Australia, Association for Computational Linguistics.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020. Is bert really robust? a strong base- line for natural language attack on text classification and entailment. In Proceedings of the AAAI con- ference on artificial intelligence, volume 34, pages 8018-8025.

---

Jing Ma, Wei Gao, and Kam-Fai Wong. 2019. Detect ru- mors on twitter by promoting information campaigns with generative adversarial learning. In The world wide Web conference, pages 3049-3055.
2021. pages 1371-1382, Online, Association for Computational Linguistics.
Tiening Sun, Zhong Qian, Sujun Dong, Peifeng Li, and Qiaoming Zhu. 2022. Rumor detection on social media with graph adversarial contrastive learning. In Proceedings of the ACM Web Conference 2022, pages 2789 2797.
Jing Ma, Jun Li, Wei Gao, Yang Yang, and Kam-Fai Wong. 2021. Improving rumor detection by promot- ing information campaigns with transformer-basee generative adversarial learning. IEEE Transactions on Knowledge and Data Engineering, pages 1-1,
Lin Tian, Xiuzhen Zhang, and Jey Han Lau. 2022. DUCK: Rumour detection social media by mod- elling user and comment propagation networks. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 4939-4949, Seattle, United States. Association for Computational Linguistics.
Ninarch Mehrabi, Ahmad Beirami, Fred Morstatter, and Aram Galstyan. 2022. Robust conversational agents against imperceptible toxicity triggers. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 2831 12847, Seattle, United States, Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cexsing Systems, volume 30. Curran Associates, Inc.
Martin Potthast, Johannes Kiesel, Kevin Reinartz, Janek Bevendorff, and Benno Stein. 2018. A stylometric inquiry into hyperpartisan and fake news. In Proceed- ings the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 231-240, Melbourne, Australia, Association for Computational Linguistics.
Gaurav Verma, Ankur Bhardwaj. Talayeh Aledavood, Munmun De Choudhury, and Srijan Kumar. 2022. Examining the impact of sharing covid-19 misinfor- mation online mental health. Scientific Reports. 12:1-9.
Reinald Adrian Pugoy and Hung-Yi Yu Kao. 2021. Unsu- pervised extractive summarization-aasee representa- tions for accurate and explainable collaborative filter- ing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Lan- guage Processing (Volume I: Long Papers), pages 2991-2 Online. Association for Computational Linguistics.
Jesse Vig. 2019. A multiscale visualization of attention in the transformer model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 37-42, Florence, Italy. Association for Computational Lin- guistics.
Svitlana Volkova, Kyle Shaffer, Jin Yea Jang, and Nathan Hodas. 2017. Separating facts from fiction: Linguistic models to classify suspicious and trusted news posts on Twitter. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), pages 647-653 Vancouver, Canada. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 188)9
Ammar Rashed, Mucahid Kutlu, Kareem Dar- wish. Tamer Elsayed. and Cansin Bayrak. 2021. Embeddings-based clustering for target specific stances: The case of polarized turkey. In Proceed- ings of the International AAAI Conference on web and social media, volume 15, pages 537-548.
Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018. The spread of true and false news online. science. 359(6380): 146-1151.
Marco Tulio Ribeiro, Sameer Singh. and Carlos Guestrin. 2016. why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135- 1144.
Ke Wang and Xiaojun Wan. 2021. TransSum: Trans- lating aspect and sentiment embeddings for self- supervised opinion summarization. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 729-742, Online. Association for Computational Linguistics.
Wojciech Samek. Thomas Wiegand, and Klaus-Robers Müller 2017. Explainable artificial intelligence: Un- derstanding, visualizing and interpreting deep Iearn- ing models. arXiv preprint arXiv: 08296.
Lingwei Wei, Dou Hu, Wei Zhou, Zhaojuan Yue, and Songlin Hu. 2021. Towards propagation uncertainty: Edge-enhanced Bayesian graph convolutional net- works for rumor detection. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 1 11th International Joint Conference Natural Language Processing (Vol- une I: Long Pupers), pages 3845-3854, Online. As- sociation for Computational Linguistics.
Yun-Zhu Song, Yi-Syuan Chen, Yi-Ting Chang, Shao- Yu Weng, and Han Shuai. 2021. Adversary- aware rumor detection. In Findings of the Associa- tion for Computational Linguistics: ACL-IJCNLP

---

Ke Wu, Song Yang, and Kenny Q Zhu. 2015, False rumors detection on sina weibo by propagation struc- tures. In 2015 IEFE s1 international conference on data engineering, pages 651-662 IEEE
way, h can approximate the embedding of a gen- crated response and be concatenated with the em- bedding of {ma) that serves as part of the en- coder's inputs without taking argmax operation. Subsequently, the gradients can be backpropagated from the rumor detection loss to train the ARG. Moreover, the generated response is attached to the source post of the thread and new edge e0,n between zn and F, is thus created.
Yong Xie. Dakuo Wang. Pin-Yu Chen, Jinjun Xiong. Sijia Liu, and Oluwasanmi Koyejo. 2022. A word is worth a thousand dollars: Adversarial attack on tweets fools stock prediction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 587-599, Seattle, United States. Association for Computational Lin- guistics.
B Experimental Setup
B.1 Datasets
Ying Xu, Xu Zhong, Antonio Jimeno Yepes, and Jey Han Lau. 2021. Grey-box adversarial attack and defence for sentiment classification. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 4078-4087, Online, Association for Computational Linguistics.
All datasets used are publicly available. Table 5 displays the statisties of RumorEval2019 (RE2019), Twitter15 and Twitter16 datasets. N, T, F, U rep- resent Non-Rumor True Rumor, False Rumor and Unverified Rumor respectively. We also calculate the statistics of the number of posts for each claim and report them as thread length in the table.
Fan Yang. Yang Liu, Xiaohui Yu, and Min Yang. 2012. Automatic detection of rumor on sina weibo. In Pro- ceedings of the ACM SIGKDD workshop mining data semantics, pages 1-7.
B.2 Implementation Details
A Adversarial Response Generator
All of our experiments are conducted on a single NVIDIA GeForce RTX 3090 GPU. We conduct 5-fold cross-validation and report the average re- sults for all datasets. The total training time re- quired for each fold on RE2019 and Twitter16 is around one hour, and two hours for Twitter15. The number of trainable parameters is around 300 mil- lion. We use the same set of hyperparameters on all datasets. Specifically, the batch size for BiTGN, ARG, and SSRA is 16, while the learning rates are set to 2 x 10-5, For DRE, the batch size and learn- ing rate are respectively 256 and 4 x 10-5 We finetune BiTGN, ARG and SSRA for 10 epochs, and DRE is trained for 50 epochs. The number of GCN layers lg in BiTGN is and the encoder and decoder of DRE both consist of 4 transformer layers. The dimension of the hidden noise z
Here, we provide more detailed formulation of the adversarial response generator. To further simulate the attack responses produced by different users in real-world scenarios, we adopt an adversarial response generator (ARG) proposed by Song et al. (2021), which is trained by adversarial learning un- der white-box settings. We initialize ARG with a BART model due to its outstanding performance on several text generation tasks. Given a conver- sation thread { 1n the goal is to generate adversarial response 2,, that makes the detector de- viate from the ground- y by maximizing the detection loss Ldet. detailed in Sec. 3.5, i.e.,
max Laeely (y",y) yY, bg
(13)
y 0 gge (Denc H (0) A'),
where h denotes the hidden representation of 2; and A' is the new adjacency that attaches 2; to source post Zo. To generate response, we con- struct the ARG by sharing the encoder Aenc with BiTGN and feeding the hidden representation of
Dataset
RE2019
Twitter15
Twitter16
of claims
446
1490
818
of N
374
205
Tf of
185
372
207
138
370
205
the last encoder layer (Le) to the decoder dde as:
# of
123
374
201
# of posts
8529
41154
18618
h O H (L))),
Max. thread length
268
304
250
(14)
24 argmax(softmaxx(P... (h))))
Min. thread length
2
1
Avg. thread length
19.
28.23
22.76
Note that fout denotes the output layer, which is tied with the input embedding layer d..- In this
Table 5: Dataset Statistics

---

RI20199
Twitter15
Twitter16
Model
mihi
Fi-T
Fi-F
m.F)
F--T
Acc.
Fi-N
Fi-T
Fi-U
RvNN
52.00
51.42
53.41
54.14
46.72
73.88
73,81
70:27
81.77
72.46
70.74
75.18
75.22
67.07
$5.49
74.25
74.09
BIGON
60.31
56.83
66.39
58.72
45.38
84.02
83,69
50.02
88.61
84,79
81.34
87.33
87.07
79.52
92.87
87.63
88.26
52.91
WEIGN
68.82
67.53
73.21
67.80
61.57
87.35
87.34
91.41
87.45
85.82
84.00)
87.16
57.09
86.32
92.79
K3.45
85.75
DUCK
69.73
02.02
72.07
71.20
63.79
86.25
86.14
87.04
88.88
84.39
84.26
86.55
86.40
79.81
93.62
86,12
86.07
BITON
70.84
70.05
74.30
69.70
66.15
87.77
87.73
89.82
$8.63
87.34
85.15
89.10
89.02
88.78
94.52
86.13
86.66
BUGCN
69.27
68.45
73:72
65.94
65.67
87.76
87.72
89,93
89.64
88.04
83,48
88.00
$8.00
86.20
93.47
$4.66
87.66
TDGCN
69.50
69.35
87.98
GCN
70.40
09:48
74.54
68.51
65.41
87.35
87.42
90.96
86.68
87.15
84.90
87.89
87.83
87.10
94. 18
84.14
85.89
Table Detailed results of rumor detection. The Pi score each class is reported. The best second best scores of the first two blocks are marked in bold /uderlined. Our BiTGN outperforms all baselines listed in the first block.
RE2019
Twitter15
Twitter16
Model
(Acc.)
(mF))
(Acc.)
(mmFi)
(Acc.
(mFFi
10
1.30 x 10
10-5
10
10
1.15 10
BIGON
9.29 x 10-3
5.32 x 10-a
3.14
10-3
1.42 10-3
1.31 x 10-1
1.30 10-1
EBGEN
4.14 x 10
4.25 x 10
1.29
1.35 x 10
2.52 x
2.39 x 10
WEIGN
2.25
1.69 10
3.32
10
3.37 10
10-1
1.37 10
DUCK-ET
3.57 10
3.37 x 10
1.13 10
1.00 10
1.09 x 10
1.12 10
Table 7: Paired student t-test between our BiTGN and other baselines for rumor detection. The p-value of both accuracy (Acc.) and macro-averaged F1 (m/l) are presented.
DRE is set to 100. Moreover, we implement our framework with Hugging Face Transformers and PyTorch. For the results of rumor detection in section 4.4, the transformer encoder of BiTGN is initialized with RoBERTa-base". For the results of adversarial attack and defense in section 4.2, the ARG shares the same encoder with BiTGN, and the overall encoder-decoder framework is initialized with BART-base". The SSRA is initialized with SaRT-baseSAMMSun
official implementation of FactCC
C Additional Results of Rumor Detection
C.1 Models with Different Backbones
We provide the results of WETGN and BiTGN using different transformer encoders, including RoBERTa with 12 self-attention layers and BART encoder with 6 self-attention layers. The results are shown in Table 8. It is obvious that both WETGN and BiTGN perform better with RoBERTa encoder, which is expected since ROBERTa contains more layers than BART encoder. Morcover, RoBERTa is pre-trained several text classification tasks, while BART is more effective on text generation tasks. We also report the rumor detection results of BiTGN after each adversarial training stage. For the first stage (BiTGN!), the inputs of the detec- tor contain the response generated from ARG, and the second stage (BiTGN*) contains only the re- sponses from original data. We can observe that the model performs better in the first stage, which indicates that the generated response in the first stage help the detector improve its accuracy.
We also follow (Song et al., 2021) to perform tree decomposition on the original datasets where each conversation thread is decomposed into several subtrees by adding each response one by one in chronological order. this way, we not only increase the amount of training data for BiTGN but also create a pseudo-ground- truth response for ARG from the last response of each subtree. For the baseline models of rumor detection, we use the official implementation for RvNN" 12 EBGCN'3 and DUCK .r For 1ETGN we implement the model architec- ture by ourselves due to the similar design. For the evaluation of factual consistency, we use the
https://uuggingface.ooooee cowroberta--boooooooo base
C.2 Detailed Detection Results
Table 6 provides the detailed results of rumor de- tection, including the F1 score for each class. The results demonstratc that BiTGN outperforms all baselines listed in the first block, demonstrating
19
11 http://wwwunnnnnnnoooo/c/iii///a base-samsum
12 httttp:://gitbbb... wom/TiiaBBann9///iCCC
17 hhtppp//giiibbb..m////llll////// com//elliingwwww/////DDCCCCC
19 http://////////oooooooooooooooaa
15 http:///ghhhhhccoo///////////aat

---

Macro F,
ASR
Macro Fr
ASR
3.6
0.6
0.7
DAS, k=3
DAS, k-3
DAS,
DAS,
0.5
0.4
DAS, k-5
0.4
DAS, k-5
0.4
0.4
who DAS, wio attack
wio DAS, w/o attack
0.3
w/ attack
DAS, attack
0.
0.2
0.4
0.6
0.4 0.0
0.2 0.6
0.
0.2
0.6
0.6
Figure 5: Effect of extract ratio o and number clusters k on RE2019 (left) and Twitter16 (right). The dashed lines represent the detection performance without DAS. The Macro-/ increases the extract ratio p decreases both datasets, and the Attack Success Rate (ASR) behaves in an opposite trend. Moreover, the number of clusters k do not influence results significantly, demonstrating the robustness DAS.
RE2019
Twitterl5
Twitter16
of transformers to independently model each con- versation thread as both stream and graph,
Model
Rackhone
Acc.
mFF
Aoc.
mFF
Acc.
m.Fi
WETGN WETGN
BART
67.03
65.59
36.53
86.40
85.82
85.71
our model achieves more favorable average per-
RoBERTa
68.82
67.53
$7.35
87.34
87.16
$7.09
BITGN!
BART
66.14
65.69
86.4
86.79
86.06
85.95
formance by utilizing only single transformer
BITGN*
BART
64.79
64.46
86
85.
85.81
83.76
branch, thereby resulting in fewer parameters.
BITGN
RoBERTa
70.84
87.77
89.10
Table 8: Performance of BiTGN and WETGN with BART and RoBERTa as transformer encoder. RoBERTa encoder improves the performance on all models.
D Adversarial Attack and Defense on Other Datasets
We investigate the robustness of DAS under dif- ferent extract ratio p and number of clusters k on RE2019 and Twitter16 datasets, as illustrated Fig. The dashed lines represent the detector's per- formance without equipping the DAS framework. Specifically, the red line and green line stands for the performance of the model with and without be- ing attacked respectively. The results of all datasets are intuitive and similar, where the Fi score in- creases as the extract ratio p decreases, and the Attack Success Rate (ASR) behaves in an opposite trend. Moreover, increasing the number of clusters to provide responses from more perspectives do not affect the model performance drastically.
its power contributed by the robust textual repre- sentations from the transformer network and the effective graph aggregation from the BiGCN com- ponent. Moreover, as discussed in section 4.4, we see that BITGN has a lower F1 score on the classes of non-rumor and truc rumor of Twitter15, which may be caused by the diverse structural informa- tion of this dataset as observed by Huang et al. (2020). To validate the significance of the detec- tion results, we perform the paired student t-test between the proposed BiTGN and each of the base- line models. The corresponding p-values for both accuracy and macro-averaged F1 scores are pre- sented in Table 7. Notably, our BiTGN not only achieves the best average accuracy and Fi scores discussed previously but also significantly out- performs models utilizing TF-IDF vectors as node features (BiGCN, EBGCN, RvNN). We notice that the significance levels for the comparisons with WETGN and DUCK-UT are relatively lower. This could be attributed to their shared utilization of transformer backbone and the notable variability in folds. Nonethcless, in comparison to WETGN, which incorporates a top-down GCN with weighted edges, our BiTGN effectively benefits from the bi-directional GCN component, even in the ab- sence of weighted edges. When compared against DUCK .UT, which employs two distinct branches
E Additional Results of DAS Framework E.1
Rumor Detection with DAS
To observe how the proposed response summa- rization framework affects the performance of ru- mor detection, we provide the detection results of BiTGN equipped with the variants of DAS with- out being attacked in Table 9. DRE! denotes DRE with the autoencoder (AE) only. In this experi- ment, BiTGN is initialized with BART-base, and the number of clusters k is set to 3, i.e., DRE ex- tracts responses, and DAS further produces ab- stractive summaries based on the 3 clusters. Firstly, we observe that both DRE and DRE can approx- imate the model's performance. Notably, DRE

---

RE:0119
Twitter15
Twitter16
Summarizer
Acc.
Fi-T
Pi-F
Fi-U
Acc.
InFi
Fi-N
Ri-T
Fi-H
Pi-U
Acc.
m.Fi
Fi-N
Fi-T
Fi-F
Fi-U
64.57
64,29
65,25
64,01
63.63
85.64
85,50
58.52
87.75
85.35
80.37
84.47
84,40
85:62
92.16
80.78
79.06
DRE (pm0.05)
60.08
59.88
62.22
60.50
56.93
83.78
83.68
85.51
86.49
83.43
79:31
82.39
82.31
$1.88
91.80
76.81
78.73
DRE (a=0.10)
60.10
64.32
58.74
57.24
87.59
83.42
79.52
62.52
82,45
82,35
92.03
78.61
DRE' (p=0.15)
60.76
60:41
62.54
60.77
57.92
84.54
84.43
$7.49
86.85
84.12
79.27
83.13
83.07
83.63
91.88
77.92
78.84
DRE (ym0.20)
61.43
61.10
63.33
62.
57.86
84.67
84.54
87.40
86.48
84.23
80.08
82.89
82.83
84.02
90.25
77.54
79.03
DRE ()m0..5)
61,88
61,41
63.53
64.25
56.45
84.95
84.85
87.42
86.62
84,64
50. 70
83.15
83.08
54.02
90.75
78.26
79.31
DRE! (p=0,50)
63.00
62.40
64.54
64.31
58.36
86.12
85.96
89.08
88.42
85.62
80.73
83.38
83.37
83.96
90.40
78.14
$1.00
DRET (pm0.75)
64.35
66.14
64.94
59.
85.
80.07
84.96
84.94
85.60
92,45
81.60
DRE ()=0.90)
64.12
63.49
65.21
64. 16
61.10
85.91
85.77
910
87.90
83.58
08
84.22
84.12
83.78
91.98
79.75
80.99
DRE (p=0.05)
60.08
59.88
62.22
60.50
56.93
83.78
83.66
85.51
86.49
83.43
79.31
62.39
$2.31
81.88
91.80
76.81
78.73
DRE (y=0.100
60.53
60.22
62.93
61.15
56.58
84.19
84.11
87.16
86.71
83.10
79.45
82.76
82.69
82.81
92.03
77.11
78.80
DRE (o:0.155)
60.31
59.97
62.15
60.03
57.72
$4.54
84.45
87.66
86.85
83.65
79.63
83.01
82, $2.95
83.36
91.88
77.71
78.84
DRE (p=0.20)
61,43
61.11
63.35
62.31
57.66
84,95
84.86
87.59
86.97
84.57
80.33
62.88
82.84
83.41
90.64
77.52
79.80
DRE (p=0.25)
61,88
61.50
63.77
64.42
56.30
85.15
85.07
857.49
87.01
85.19
80.59
83.15
85,
83.98
91.11
78,06
79.17
DRE (p=0.55)
62.10
61.70
64.02
61.09
59.97
$4.95
84.98
57.65
87.04
84.47
80.37
82.89
82.84
82.50
90.21
77.48
81.15
DRE (ym0.75)
62.53
84.57
79.05
84.22
84.23
85.51
80.96
DRE (p=0.90)
61.88
61,46
63.75
62. 28
58.35
84.95
84.83
88.00
87.10
84.61
79. 60
83.62
83.61
83.65
91.18
78.71
80.92
DRE (=01.00)
59.35
60.50
84.54
84.00
83.11
83.34
78.26
DAS (p=0.05)
61.66
60.99
64.81
61.22
56.93
81.31
81.13
82,20
84.88
80.50
76.97
80.93
$0.73
78.42
90.60
76.22
77.68
DAS (p=0.110
62.56
61.88
65.87
61.08
58.70
81.72
81.53
82.20
85.20
81.05
77.65
81.42
81.25
79.19
91.05
77.47
77.30
DAS (p=0.15)
64.10
61.51
59.34
78.15
81.11
77.12
78.24
DAS (p=0.20)
63.00
62.29
65.35
62.98
58.54
82.75
82.60
$4.39
84.82
$1.80
79.37
81.05
$0.99
80.38
88.32
76.29
78.95
DAS (p=00.25)
63.90
63.09
67.15
64.14
57.97
82.41
82.26
833554
85.29
81.60
78.61
82.28
82.19
80.97
89.54
78.72
79.53
DAS (p=0.50)
62.54
63.28
57.50
$3.09
82.98
84.37
81.83
79.18
82.10
79.71
89.51
79.98
79.22
DAS ()-0.775
63.68
62.94
67.14
61.87
59 80
85.09
82.98
86.02
86 98
81.12
77.79
K2.51
82.43
81.25
90.99
77. 34
80.15
DAS (p=00.90)
63.67
62.73
66.67
61.55
59.96
83.09
82.94
84.91
86.85 89
81.66
78. 30
83.25
83,19
81.15
90.41
79.90
81.28
DAS ()=1000)
64.34
63.44
67.31
63
40
59.60
83.16
83.06
84.84
87.59
81.31
78.53
82.64
82,54
82.07
91.31
77.42
79.36
Table 9: Rumor detection results of BiTGN using different summarizers without being attacked. The results of DRE!, DRE, and DAS are demonstrated in the upper, middle, and lower sections, respectively. DRE' represents DRE with the autoencoder (AE) only. The number of clusters l is set to 3. Both DRE and DRE can approximate the detection results of BiTGN without summarizers, and DAS only slightly degrades the performance.
even scores higher on the Twitter15 and 1witter16 datasets, potentially due to the filtering process, which identifies and removes noisy responses that could degrade the model's performance. Moreover, DRE achieves comparable results even if only responses are selected, indicating that the extrac- tor can effectively capture the representative re- sponses from each conversation thread. As such, the extracted responses can be used to interpret the model's behavior. Next, for all summariz- ers, the performance rises as the extract ratio p increases and saturates at different ratios on differ- ent datasets, which suggests that the representative responses could be potentially excluded if we fil- ter out too many of them. Lastly, we also find that DAS slightly degrades the detection perfor- mances, possibly due to the distribution shift of abstractive summaries as the abstractor is tuned under self-supervised settings without fine- -raiied ground-trune truth labels. Furthermore, the unsatisfy- ing quantity of data also indicates that there is still room for improvement. We consider enhancing the text quality under self-supervised settings as a potential avenue for future research.
E.2 Human Evaluation
In this section, we further provide the settings and additional results of human evaluation. We recruit 100 human readers for both parts of the human evaluation. We additionally insert a trick question in part A to validate the answer quality of each participant, and each participant will be paid $5 if he / she passes the trick question. To verify the consistency of the collected results, we calculate the Fleiss' Kappa to evaluate the inter-rater reli- ability of the human evaluation. For part A, we calculate the Fleiss' Kappa to assess the readers' agreement toward rating our model more favorably than the baseline, and the score is 0.3321. In part B, for readers that make predictions based on the re- sponses and summary, we obtain the Fleiss' Kappa scores of 0.5341 and 0.4225, respectively. Follow- ing the criteria outlined by Landis and Koch (1977), these values indicate fair and moderate agreement among participants in part A and B. It's important to note that this level of agreement has been es- tablished as reliable in prior research studies (Cao and Wang, 2021: Chen et al., 2021). In part A, we select SSRA-LOO as the baseline and compare its informativeness with SSRA-k -ae-----meess (k=3). Since

---

Number of predictions based on Response / Summary
Ground-Truth
False
True
Resparse
20
Summany
20
Figure 6: Visualization of the number of predictions based on responses summary for each sample human evaluation part B. Ground-truth label for each sample is marked with "+". . In most cases, predictions based on the responses and the summaries are highly correlated, demonstrating the interpretability of the summaries.
SSRA-LOO generates only one summary by de- fault, randomly divide each conversation thread into 3 groups and make SSRA-LOO generate one summary for each group for a fair comparison. We ask the participants to rate each set summaries based on the following scoring strategy:
summaries deny the rumor, but the readers not required to accept all the utterances and can make
their decisions based on their intuition after reading the provided information. We visualize the predic- tions based on the responses and summary in Fig. 6. Notably, a higher number of individuals accu- rately predict the veracity (Pyt > 0.5) of 15 and 17 samples based on the responses and summary. respectively. Despite the slightly lower average accuracy of the summary, shown in Fig. the results still indicate its effectiveness providing social media users with essential information from the responses. Additionally, we observe a high cor- relation berween the predictions obtained from the responses and summaries in most cases. This em- phasizes the interpretability of the summaries, identify the crucial information that the de- tection models focus when making predictions.
Score None of the three summaries accurately capture the information from the responses, and the summaries may repeat the same information or contain unrelated information.
Score 2:
Only one summary captures the infor- mation from the responses, but the other ones may be incomplete, inaccurate, or repetitive.
Score 3: Most of the summaries accurately cap- ture the information from the responses, but some important perspectives may be missing, or some information may be repeated unnecessarily.
E.3 Generation Examples
Score 4: All three summaries accurately capture the information from the responses but may not fully cover all important perspectives or provide nuanced understanding of the issue.
We demonstrate more examples by DAS in Ta- ble 10, and 11. QID corresponds the Question ID of part B human evaluation, as shown in Fig. 6. We provide the source post, responses, and both extrac- tive and abstractive summaries for each example. The responses are arranged in different clusters (i.e., Cluster 1, Cluster 2, and Cluster 3) and chrono- logical order (i.e., [1]. [2], ...). Specifically, we highlight the crucial information of the responses captured by the summaries with different colors for each cluster. The results show that both extractive and abstractive summaries can capture essential viewpoints from different perspectives. Moreover, the responses in the same clusters contain similar information, indicating the clustering mechanism
Score 5: All three summaries accurately capture the information from the responses, and the sum- maries provide comprehensive and nuanced understanding of the issue. The summaries cover diverse perspectives and avoid repetition.
In part B, select 20 samples from RE2019 and Twitter15 datasets, with 5 true rumors and false ru- asked
mors from each of them. The participants determine whether a source post true or false based on all its responses or a set of summaries. A key idea is to observe whether the responses or

---

in DAS can indeed identify the underlying aspects in the diverse responses of a conversation thread. The summaries can not only explain predictions from rumor detection models but also benefit social media readers and make them quickly understand the public's 's opinions toward specific events.

---

QID
Content
Source Post (False Rumor): a claim that #obama used the #shutdown to scuttle the amher alert system reveals an ignorance about amber alerts. URL
Responses (Cluster 1) [2]: "@namml : Claim #Ohama used #shutdown to scuttle AmberAlerts reveals ignorance aht AmberAlerts URL." @ namme [5]: RINO!!! RT @ @ammel A claim that #Ohama used the #shutdown to scuttle the
Amber Alert system reveals ignorance about Amber Alerts. [6]: @ namel Thebn what was he doing when he shut it down? [9]: @ namel @ namee you retweeted ignorance today. "Amber alent " False [10] @namel No doesn't I It shows how ignorant ppl ure for believing stupid S % #SHUTDOWN (Cluster 2) [1]: @namel Just got asked to "inform" someone about this since he/she can't 't hothered inform self. URL. (Cluster [3]: @namel @namee Perhaps, but a smart administration wouldn't have put that notice on the site. have no sympathy. [4]: I @name4 That's not the claim. The claim is that Obama trying to scare people by appearing shut down sites, [7]: @namel @ name5
really sick and tired of seeing more of this fake news finding traction; how stupid are these readers? [8]: @ name1 namee6 Why care at all?? Maybe the little scamps wandered off look for some food since their SNAP was cut. Just guess. [11]: @ @name? I'm willing to bet there's high probability that the people who helieve this might also helieve Ohama is Kenyan [12]: @namel @amee8 These lies about the #GOPShutdown are not isolated. They're part of RNC strategy distraction URI.
Extractive Summary [1]: @ @namel No it doesn't It shows how ignorant ppl are for believing stupid S # % #SHUTDOWN [2]: @ameel I Just got asked to "inform" someone about this since he/she can't be bothered to inform self. URL [3]: @namel @name4 Perhups, but smart administration wouldn't 't have put that notice the sitc. 1 have sympathy.
Abstractive Summary [1]: Claim #Obama used #shutdown to scuttle Amber Alerts reveals ignorance abt Am- berAlerts [2]: I don't know how to inform someone about this since he/she can't 't be bothered to inform self. [3]: That's 's not the claim. The claim is that Obama is trying to scare people by appearing to shut down sites.
Source Post (True Rumor): krispy kreme hull advertising kkk wednesday. don't know. do not know. URL
Responses (Cluster 1) [1]: @namel Via Burlingame. URL [4]: @namel @name2 Is that real ad? [6]: @namel sprinkled with WHITE POWDER? [9]: Anybody could make that... @ @namel @ name3 [11]: @ @namel Ack! [12]: @ name4 Hull, we need to talk... [16]: @ namel [17]: @ namel seriously, how could someone had been so stupid. [19]: @namel| wonder they use... URL (Cluster 2) [2]: "namel: Krispy Kreme hull is advertising KKK Wednesday. don't 't know. I do not know, URL" #Fail [3]: @namel @name2 Its a company born in the dirty south, so0o? [5]: Wow! RT @ namel: Krispy Kreme hull advertising KKK Wednesday. don't 't know. I do not know. URL" [14]: namee @amel Which is really good. Unfortunately, the mockery will continue until morale improves. [18]: @ @namel @ name6 Krispy Kreme is headquartered in the American South, (Winston Salem, North Carolina) sooo... [20]: @ @namel @ name? The Kicker is that this was advertised to children Krispy Kreme Klub, teaching racism and poor spelling all in one. (Cluster 3) [7]: @ @namel Hi, know we got wrong & wholeheartedlyy 11 upologise. We're taking steps make sure doesn't 't happen again [8]: @namel They'r 'r dull in Hull and the Islc of Mull is scething with discontent. [10]: @namel I GG Hull, glad know the 2017 city of culture trying to be inclusive of all groups [13]: @namel Like the article said. It was poor choice of the play on the word Club (*spelled Klub) Their intentions were good though. [15]: @ name5 @ namel I you't 're going to take step to make sure you don't 't create any more sales events with unbelievably racist names? Okay.
Extractive Summary [1]: @namel I Via Burlingame. URL [2]: Wow! RT "@namel: Krispy Kreme hull is adver-
tising KKK Wednesday. don't 't know. I do not know, URI." [3]: @ namell GG Hull, glad to know the 2017 city of culture is trying to be inclusive of all groups
Abstractive Summary [1]: That's so stupid!!! [2]: Krispy Kreme hull is advertising KKK Wednesday, don't know. I do not know, [3]: I'm sorry to hear this, but it was a poor choice of the play on the word Club (*spelled Klub).
Table 10: Generated examples of DAS (k=3). QID corresponds to the Question ID in Fig. 6.
The responses are arranged in different clusters and chronological order. Key information captured by summaries is highlighted with different colors for each cluster. The responses within the same cluster deliver similar information, and the produced summaries can effectively capture essential information from the responses.

---

QID
Content
Source Post (False Rumor): new. leaked phone call between rebel leader & russian intel agent: cossacks Shot down #mh177 URL URL
Responses (Cluster 1) [5]: @ @namel @ name2 who leaked this? Do you check sources? Have you verified anything?
[6]: "Fuck them. They should not fly, we are at war here." RT @ @name1: I: NEW. Leaked phone call between rebel
leader& & Russian intel agent #MH17 [7]: RT @ammel: NEW. Leaked phone call between rehel leader & Russian intel agent: "Cossacks" shot down #MH177 URL ... [8]: 1/2 RT @ammel Leaked phone call/rebel leader Russian intel agent: "Cossacks" shot down #MH17. URL URL [12]: @namel It not seems as a reliable source. Now all are
fabricating they own versions. [14]: @ n @ @fname3 careful what you helieve; way too much propaganda and lies out there [15]: @ namel @ name4 NEW. Leaked phone call between rebel leader & Russian intel agent: "Cossacks"
shot down #MI117 URL [18] @namel "indonesian student" ??? my poor fellow countrymen.... ... #dlamn you terrorist! (Cluster 2) [4]: @namel such violence and hate but will be celebrating Mandela's Birthday tomorrow with acts of giving and love in South Africa #peace [17]: @namel Another Hollywood story is making :-) CIA should make story bank. Poor Hollywood is struggling for JanesBond films. @name5 [20]: @ @namel Jim, Jim, Jim. shame you. It was a hoax of sorts and you promoted it Opps there goes our credibility again Too bad @ name6 (Cluster 3) [1]: @ @ this is beyond insane... [2]: @ nameel Leaked by whom? Or don't 't you bother verifying sources? [3]: #5
13
RT @namel NEW. Leaked phone call between rebel leader & Russian URL. URI 19]: @namel [10]: @ @amell checked that? [11] namel @nmme? name8 [13]:
AE E à: 4E [ 2) jif B6
= + -) 2 h L f=]
namel @amme) Thus it begins... [16]: @9name10) 10 @ namell
@ @namel おb 1) + 9 + 5 R 2 3 TrV bb
5 [19] @namel @name5 what is the proof
Extractive Summary []]: NEW. Leaked phone call between rebel leader & Russian intel agent: Cossacks" shot down #MH17 URL [2]: @namel Jim, Jim, Jim, shame you. It was hoax of sorts and you promoted Opps there goes credibility again Too bad @ @name6 [3]: @ naaell name? Thus begins...
Abstractive Summary [1]: RT. NEW, Leaked phone call between rebel leader & Russian intel agent: "Cossacks"
shot down #MH17 [2]: This a hoax, it hoax of sorts and you promoted Shame on you [3]: what the proof??
Source Post (True Rumor): microsoft is reportedly buying 'minecraft" developer mojang for $2 billion URL
Responses (Cluster 1) [2]: @ namel Good job, @name2 [8]: @ namel @name3 NOOO [12] @namel that's one way to get it onto Windows Phone. [15]: @namel Pls no. [14]: @namel NOOO!!! !!! [18]: @namel don't do it mojang!!! ... [19]: @namel oh god why.. (Cluster 2) [T]: @namel For some reason don't believe that. They
turned down offers before, so i'm sure they will again. [3]: "@namel: Microsoft is reportedly buying 'Minecraft"
developer Mojang for $2 billion URL" [5]: "@namel: Microsoft is reportedly buying 'Minecraft' developer Mojang for $2 billion URL" @name4 [6]: "@ @namel: Microsoft reportedly buying 'Minccraft' developer Mojung for
billion URL" wauw! @ @name55 [7]: @namml: Microsoft is reportedly buying 'Minecraft' developer Mojang for $2
billion URL.? [10]: " nammc: Microsoft is reportedly buying 'Minecratt' developer Mojang for $2 billion URI." whoa [13]: "@namel: Microsoft is reportedly buying 'Mincrrft'' developer Mojang for $2 billion URL" Wat? [20]: @ namee "@ 1@mmm Microsoft is reportedly buying 'Minecraft' developer Mojang for $2 billion URL" (Cluster 3) [4]: @name7 @name8 @namml better huy minccraft now before it's off the psn store [9]: @ name9 wnoott?? Have you ever played Minecraft? That game will cat up your time I swear and babu objectives just dig up sand and create [11]: @ @amee7 @name8 @ @namel lol already on there. [16]: @ name 10 @ @namel @ @ @name2 From what I'm told, Notch walked away from Mojang. He's doing things on his own now. [17]: @name II @ namel yeah microsoft
16
@ name7 @ name8
Extractive Summary [A]: @namel I Pls no. [2]: "@ namel: I: Microsoft is reportedly buying 'Minecraft' developer Mojang for 52 billion URL" whoa [3]: name7 @name& @namel lol already on there.
Abstractive Summary [1]: NOOO!!!.. [2]: Microsoft is reportedly buying 'Minecraft' developer Mojang for
billion [3]: think Minecraft is great game,
Table 11: Generated examples of DAS (k=3), QID corresponds to the Question ID in Fig, 6. The responses are arranged in different clusters and chronological order. Key information captured by summaries is highlighted with different colors for each cluster. The responses within the same cluster deliver similar information, and the produced summaries can effectively capture essential information from the responses.


Reviews:
[{"review_id": "5uSPpwZtKn", "reviewer": "Reviewer_4y82", "summary": "", "questions": "A: It is usually the latent space representations that are clustered. Why is it that the output representations in DAS encoder-decoder are clustered?\n\nB: Can we find a way to simply or clarify Fig. 2?\n\n\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 4y82, \\\nWe sincerely appreciate your comments and suggestions. We have revised the mentioned parts in our paper and would like to clarify several points as follows.\n\n1. **The complexity of the system is oversold which affects the readability of the paper.** \\\nThank you for your suggestions. Figure 2 may convey an excessive level of detail. Accordingly, we have made necessary modifications to the figure as follows: Firstly, for both the overall framework (upper left) and the rumor detector architecture (upper right), we have simplified the notation of input from $x_1, …, x_5$ to $x$. Additionally, we have added two blocks to represent the input and output embedding layers. For the DAS framework (lower), we preserve $h_1, …, h_6$ to illustrate the clustering mechanism performed on multiple responses. To enhance the readability, we have also included a dashed arrow connecting $h_1, .., h_6$ in the lower figure (DAS) to emphasize that the encoder-decoder within the extractor is solely employed to filter the responses. The revised figure can be found here: https://anonymous.4open.science/r/EMNLP2023-2B9A/model_overview.png\n2. **Sentences closer to each other in a latent space are more useful to be summarized jointly.** \\\nHere, we want to clarify the motivation for summarizing sentences closer in a latent space. To be more precise, our corpus does not include any summary labels, and the challenge in self-supervised summarization is how to construct an appropriate pseudo-summary. In this paper, instead of randomly selecting one response from all responses as the pseudo summary repeatedly, we propose to choose responses from clusters as the self-supervised learning signal. In a social media conversation thread, the diverse utterances of different users increase the difficulty of summarizing a specific aspect from any other perspective. Consider an extreme scenario where a conversation thread includes mostly supportive opinions and a single denying response toward the source post. If the denying one is selected as the pseudo-summary, it is unreasonable and injects substantial noise and difficulty in training a summarizer. Therefore, leveraging a clustering mechanism to aggregate similar utterances becomes pivotal, as this strategy could enhance the reliability of the pseudo-summary construction for providing diverse viewpoints within a conversation thread.\n3. **Not clear why this method is only applied or applicable to response summarization. Can’t this framework be extended to most domains?** \\\nWe choose to develop this method for the rumor detection task for two main reasons. Firstly, given the prevalent use of social media nowadays, creating malicious responses to mislead model predictions becomes remarkably effortless, which leads to severe security concerns for detection models. Secondly, considering that readers can be easily influenced by one-sided information, enhancing the interpretability of detection models can not only aid readers in comprehending model behavior but also encourage the public to thoroughly consider various viewpoints before validating unverified rumors. Our proposed method aims to improve the detector's robustness by filtering out malicious responses and provide interpretability by summarizing different opinions in a conversation thread. As such, these methods can be generalized to other tasks that rely on various user feedback to make the prediction. For instance, in the context of review-based recommendation systems, we could employ the concept of response summarization to enhance attack resistance against malicious reviews by filtering and summarizing the user reviews of a given product. We will include the above discussion in our revision to shed light on future research.\n4. **Why do we cluster the output representations of DAS?** \\\nIt is worth noting that we employ the original token embeddings for clustering instead of the latent representations from intermediate layers or the output representations generated by the encoder-decoder within the DAS. The encoder-decoder framework is adopted to detect the anomalies based on the reconstruction error. After filtering out the anomalous responses, we then use the original token embedding for response clustering. While it is applicable to use the latent representation from the encoder-decoder model for clustering, we found that using token embedding is sufficient to yield successful results in consolidating responses with similar content. Therefore, we choose to use the token embedding to keep our approach succinct. Furthermore, we have modified Figure 2 to better illustrate our approach.\n5. **Can we find a way to simplify or clarify Figure 2?** \\\nThank you for your advice. We have made modifications to Figure 2 as mentioned above to enhance its simplicity and clarity.\n6. **Examples up font will improve the readability of the paper which otherwise comes across as dense.** \\\nThanks for pointing this out, we will try to scale up the font size of the examples in the new version of our paper.\n7. **Lines 284-291 (ARG) need clarification in the Methods.** \\\nThanks for the comment. We did not include the details of ARG in the submitted version due to the space limitations. In the revised version, we will provide more details in the appendix as follows: \\\n\"*To further simulate the attack responses produced by different users in real-world scenarios, we adopt an adversarial response generator (ARG) proposed by [1], which is trained by adversarial learning under white-box settings. We initialize ARG with a BART model due to its outstanding performance on several text generation tasks. Given a conversation thread $\\\\{x_i\\\\}^n_{i=0}$, the goal is to generate an adversarial response  $x_n^\\ast$ that makes the detector deviate from the ground-truth $\\mathbf{y}$ by maximizing the detection loss $L_{det}$, detailed in Sec. 3.5, i.e.,*\n\n    $max_{\\mathbf{h_n^\\ast}}(L_{det}(\\mathbf{y}^\\ast, \\mathbf{y}))$, \\\n    $\\mathbf{y}^\\ast=\\theta_{gcn}(\\theta_{enc}([\\mathbf{H}^{(0)} || \\mathbf{h_n^\\ast}]), \\mathbf{A}')$, \n\n    *where $\\mathbf{h_n^\\ast}$ denotes the hidden representation of $x_n^\\ast$ and $\\mathbf{A'}$ is the new adjacency that attaches $x_n^\\ast$ to source post $x_0$. To generate a response, we construct the ARG by sharing the encoder  $\\theta_{enc}$ with BiTGN and feeding the hidden representation of the last encoder layer $\\mathbf{H}^{(L_{e})}$ to the decoder $\\theta_{dec}$ as:*\n\n    $\\mathbf{h_n^\\ast}=\\theta_{dec}(\\mathbf{H}^{(L_e)})$, \\\n    $x_n^\\ast=argmax(softmax(\\theta_{out}(\\mathbf{h_n^\\ast})))$.\n\n    *Note that $\\theta_{out}$ denotes the output layer, which is tied with the input embedding layer $\\theta_{in}$. In this way, $\\mathbf{h_n^\\ast}$ can approximate the embedding of a generated response and be concatenated with the embedding of $\\\\{x_i\\\\}^n_{i=0}$ that serves as part of the encoder's inputs without taking $argmax$ operation. Subsequently, the gradients can be backpropagated from the rumor detection loss to train the ARG. Moreover, the generated response is attached to the source post of the thread and a new edge $e_{0,n}$ between $x_0$ and $\\widehat{x}_n$ is thus created.*\"\n\n[1] Yun-Zhu Song, Yi-Syuan Chen, Yi-Ting Chang, Shao-Yu Weng, and Hong-Han Shuai. 2021. Adversary-Aware Rumor Detection. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1371–1382, Online. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A: Few papers in this track attempt to formalize the task and write out the entire methods in a clear and concise mathematical framework.\n\nB: I really appreciate the ablation study in Table 4 claiming that the GCN complex is necessary to boost performance (although very little compared to the second strongest baseline). \n\nC: I find the pseudo-summary setup in the training of SSRA-LOO to be inspired and useful and I look to use that in my own future work.\n", "reasons_to_reject": "A: The complexity of the system is oversold which affects the readability of the paper. Consider Figure 2: $x_1, \\dots, x_5$ can be simply written as $x$, and a small block can imply encoding $x \\rightarrow h$. DAS finds two summaries: \"extractive\" - from the text, \"abstractive\" - summarizes in own words. This is facilitated by BART.\n\nIn simple terms, the rumor detector uses summaries from the text to predict whether it is a rumor or not. The novelty appears to be in the method in which these summaries are constructed; Argument being that the points corresponding to sentences closer to each other in a latent space are more useful to be summarizes jointly. \n\nB: In the first read, it wasn't clear to me why this method is only applied or applicable to response summarizes. Can't this framework be extended to most domains?", "questions_to_authors": "A: It is usually the latent space representations that are clustered. Why is it that the output representations in DAS encoder-decoder are clustered?\n\nB: Can we find a way to simply or clarify Fig. 2?\n\n\n", "paper_topic_and_main_contributions": "The paper proposes a novel framework in the interpretation of rumors and conspiracy theories on social media caled Defend and Summarize or DAS. The key principle here is that by focusing the attention on responses and adding summarization modules of varying levels of abstraction, we can get multiple perspectives to be able to detect rumors and also understand why they were rumors in the first place.\n\nThe model architecture contains a Transformer backbone, along with a Graph Convolutional Head and the performance of DAS is demonstrated across popular rumor corpuses that have been previously released.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None come to mind."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "I might have missed a few spots. Please find the suggestions below:\nA: Examples up front will improve the readability of the paper which otherwise comes across as dense.\nB: Lines 284 - 291 need clarification in the Methods."}}, {"review_id": "JJQfC6t1BS", "reviewer": "Reviewer_PMQb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer PMQb, \\\nWe appreciate the comments and suggestions provided in your review. We have revised the mentioned parts in our paper and would like to clarify several points in the following.\n\n1. **The assumption of extractor and abstractor (“responses with similar stances or viewpoints should lie closer in the embedding space”) is made without previous literature support or experiment support.** \\\nThank you for highlighting this matter. We have provided the following descriptions and citations to support this assumption in the revised version. \\\n“*The design of DAS follows the idea that responses with similar stances or viewpoints should lie closer in the embedding space. This concept is substantiated by prior studies [3, 4], which showcase that various standpoints of political opinions on Twitter can be well partitioned into distinct clusters based on the embedding representations. This characteristic could enhance summarization with more structured and comprehensive information.*”\n2. **Certain claims are made without justification (section 3.2 “transformer encoders are robust” is claimed without any literature or data support).** \\\nThanks for the comment. In the original paper, we aim to emphasize the importance of robust textual representations. We have added the following descriptions and citations in section 3.2 to enhance the claim in the revised version. \\\n“*Previous studies have shown that Transformer-based models are more robust to out-of-distribution data [1] and adversarial attacks [2] compared to conventional models such as CNN and RNNs.*”\n3. **The captions for the figures are not informative.** \\\nThank you for pointing this out. We have provided more detailed information for each figure in the revised version as follows. \\\n“*__Caption of Figure 1__: Three examples for the predicted probability of each class with respect to the responses on the Twitter15 dataset. Curves with their face colored represent the ground-truth labels for their source post. Critical responses that result in prediction shifts larger than 0.5 are marked with a red circle. \\\n__Caption of Figure 2__: Overview of our proposed framework (upper left). The rumor detector BiTGN (upper right) is trained to predict the veracity of each source post. The response summarizer DAS (lower) preemptively filters out attack responses generated by the response generator. It then organizes the remaining responses into $k$ clusters and produces both extractive and abstractive summaries for each cluster accordingly. During inference, the detector makes predictions based on the source post and the summaries. \\\n__Caption of Figure 3__: Effect of extract ratio $\\rho$ and number of clusters $k$ on Twitter15. The dashed lines represent the detection performance without DAS. The Macro-F1 (left) increases as the extract ratio $\\rho$ decreases on both datasets and the Attack Success Rate (right) behaves in an opposite trend. Moreover, the number of clusters $k$ does not influence the results significantly, demonstrating the robustness of DAS. \\\n__Caption of Figure 4__: Human evaluation of generated summaries. In part A, our SSRA-$k$-means model can generate more informative response summaries compared to SSRA-LOO. In part B, human predictions based on either responses or summaries can achieve comparable accuracy, which demonstrates the interpretability of the summaries.*”\n4. **“Improve detector vulnerability” $\\rightarrow$ “reduce detector vulnerability” or “improve detector robustness”** \\\nThanks for the correction. We have modified the descriptions accordingly in the revised version.\n\n\n[1] Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. 2020. Pretrained Transformers Improve Out-of-Distribution Robustness. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2744–2751, Online. Association for Computational Linguistics. \\\n[2] Jin, Di, et al. \"Is bert really robust? a strong baseline for natural language attack on text classification and entailment.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 05. 2020. \\\n[3] Darwish, Kareem. \"Quantifying polarization on twitter: the kavanaugh nomination.\" Social Informatics: 11th International Conference, SocInfo 2019, Doha, Qatar, November 18–21, 2019, Proceedings 11. Springer International Publishing, 2019. \\\n[4] Rashed, A., Kutlu, M., Darwish, K., Elsayed, T., & Bayrak, C. (2021). Embeddings-Based Clustering for Target Specific Stances: The Case of a Polarized Turkey. Proceedings of the International AAAI Conference on Web and Social Media, 15(1), 537-548.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "•\tThis paper clearly identifies two key drawbacks of existing rumor detectors, lack of robustness against critical responses and lack of interpretability.\n•\tThe experiment results are complete and can support the claims. Overall, the experiments seem to be reproducible. The proposed method outperforms existing methods.\n•\tOverall, this paper tries to solve important issues associated with rumor detection, and provides insights for future work.\n", "reasons_to_reject": "•\tThe response extractor and abstractor is built upon the assumption that “responses with similar stances or viewpoints should lie closer in the embedding space.” However, this assumption is made without any previous literature support or experiment support. \n•\tCertain claims are made without justification. For example, in section 3.2, the authors claim transformer encoders are robust without any literature or data support.\n•\tThe captions for the figures are not informative. Therefore, the figures are hard to comprehend.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper aims to solve two challenges: (1) improve the robustness of rumor detection models against critical responses (2) improve the interpretability of deep rumor detection models. To solve these two challenges, the authors propose a rumor detection model called Defend-And-Summarize (DAS) that is robust against malicious responses. Additionally, DAS can provide explanations to its prediction results with extractive and abstractive summaries. Their experiment results on three public datasets show that DAS is resilient against attacks and can generate multi-perspective explanations. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The authors mentioned “to improve detector vulnerability” at multiple places (e.g. line 92). I think the authors meant to say “reduce detector vulnerability” or “improve detector robustness”."}, "extra_text": {}}, {"review_id": "ox78s3WNdG", "reviewer": "Reviewer_v8Le", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer v8Le, \nWe really appreciate the detailed and insightful comments. We have followed the suggestions to revise the paper accordingly. Furthermore, we would like to clarify several points as follows.\n\n1. **More recent fake news datasets at \\*ACL conferences beyond 2019** \\\nFirstly, we would like to point out that our work is centered around the task of rumor detection rather than fake news detection. The distinction between these two tasks lies in their data sources. Rumor detection primarily draws data from social media platforms, while fake news detection focuses more on analyzing formal news content. Moreover, the intricate nature of social media adds a layer of complexity to data formatting, including structural relationships among responses and the data collection process itself. To the best of our knowledge, BEARD [6] is the latest and only rumor dataset released at *ACL conferences after 2019. However, this dataset focuses on early rumor detection and does not contain structural information for each conversation thread. Furthermore, recent studies [3, 4, 5] highlight that RE2019, Twitter15, and Twitter16 are the most frequently used datasets for rumor detection. We therefore follow the literature to benchmark our methods on these datasets.\n2. **Consider non-graph baselines for rumor detection.** \\\nAfter several works demonstrate outstanding rumor detection performance by incorporating the graph structure of conversation threads [1, 2], recent studies typically design graph-based methods to strive for better results. For example, [2] showcases that GCN serves an important role in rumor detection compared to text-based detectors such as RNN-based models. Moreover, we would like to clarify that we have also removed the GCN component in the ablation studies (section 4.4 and Table 4). This ablation can also serve as a non-graph transformer-based baseline similar to the design of [10]. The results manifest the necessity of utilizing GCN in the rumor detection task, as removing GCN from both directions yields an average performance drop of approximately 0.7%.\n3. **In section 4.3 “Interpret Predictions with Summary”, better connecting summary with interpretability in the writing is key to motivating the proposed approach.** \\\nThank you for the insightful suggestion. Our primary objective in the discussion of human evaluation part B is to clarify the interpretability of the summaries as human decisions. The results illustrate that decisions made based on both the responses and summaries yield an approximate accuracy and a Pearson correlation of 0.54. This characteristic is further highlighted through case studies, where the summaries effectively capture evidential information from the responses. This aids readers in substantiating the authenticity of the source post. Concerning the interpretability of model decision-making, we can refer to Table 8 in the appendix. This table demonstrates a comparable level of accuracy when using summaries compared to using responses as inputs for the detector, demonstrating similar properties as in the human evaluation. We highly agree that the relationship between summaries and interpretability serves as an important aspect of future research. We have taken your suggestions into account and revised the descriptions in accordance with the updated version.\n4. **Highlight the evaluation metric of Fig. 3 in the caption.** \\\nThe evaluation metrics used in Figure 3 are macro-$F_1$ and attack success rate (ASR) for the left and right figures respectively. Specifically, the metrics are marked on the top for both figures. We will add the metrics to the caption for further clarification as follows. \\\n“*__Caption of Figure 3__: Effect of extract ratio $\\rho$ and number of clusters $k$ on Twitter15. The dashed lines represent the detection performance without DAS. The Macro-F1 (left) increases as the extract ratio $\\rho$ decreases on both datasets and the Attack Success Rate (right) behaves in an opposite trend. Moreover, the number of clusters $k$ does not influence the results significantly, demonstrating the robustness of DAS.*”\n5. **Provide more information about human evaluation in the Appendix.** \\\nWe have provided the details of human evaluation in the appendix according to your suggestions as follows. \\\n“*We recruit 100 human readers for both parts of the human evaluation. We additionally insert a trick question in part A to validate the answer quality of each participant. Each participant will be paid \\$5 if he/she passes the trick question. To verify the consistency of the collected results, we calculate the Fleiss' Kappa to evaluate the inter-rater reliability of the human evaluation. For part A, we calculate the Fleiss' Kappa to assess the readers' agreement toward rating our model more favorably than the baseline, and the score is 0.3321. In part B, for readers that make predictions based on the responses and summary, we obtain the Fleiss' Kappa scores of 0.5341 and 0.4225, respectively. Following the criteria outlined by [7], these values indicate a fair and moderate agreement among participants in part A and B. It's important to note that this level of agreement has been established as reliable in prior research studies [8, 9].*”\n6. **Provide statistical significance tests for Table 4.** \\\nAll of our experiments are conducted with 5-fold cross-validation including the results in Table 4. We have provided the results of significance tests in the revised version as follows. \\\n“*To validate the significance of the detection results, we perform the paired student t-test between the proposed BiTGN and each of the baseline models. The corresponding $p$-values for both accuracy and macro-averaged $F_1$ scores are presented in the following table.*\n\n    |Model|RE2019||Twitter15||Twitter16||\n    |-----|---|---|-----|----|-----|----|\n    |     |$p$ (Acc.)|$p$ (m$F_1$)|$p$ (Acc.)|$p$ (m$F_1$)|$p$ (Acc.)|$p$ (m$F_1$)|\n    |RvNN |$1.29\\times10^{-4}$|$1.30\\times10^{-4}$|$6.79\\times10^{-5}$|$7.60\\times10^{-5}$|$1.14\\times10^{-4}$|$1.15\\times10^{-4}$|\n    |BiGCN|$9.29\\times10^{-3}$|$5.52\\times10^{-3}$|$3.14\\times10^{-3}$|$1.42\\times10^{-3}$|$1.31\\times10^{-1}$|$1.30\\times10^{-1}$|\n    |EBGCN|$4.14\\times10^{-3}$|$4.25\\times10^{-3}$|$1.29\\times10^{-3}$|$1.38\\times10^{-3}$|$2.52\\times10^{-2}$|$2.39\\times10^{-2}$|\n    |WETGN|$2.25\\times10^{-1}$|$1.69\\times10^{-1}$|$3.32\\times10^{-1}$|$3.37\\times10^{-1}$|$1.27\\times10^{-1}$|$1.37\\times10^{-1}$|\n    |DUCK$_{\\lnot\\text{UT}}$|$3.57\\times10^{-1}$|$3.57\\times10^{-1}$|$1.13\\times10^{-1}$|$1.00\\times10^{-1}$|$1.09\\times10^{-1}$|$1.12\\times10^{-1}$|\n\n    *Notably, our BiTGN not only achieves the best average accuracy and $F_1$ scores as discussed previously but also significantly outperforms models utilizing TF-IDF vectors as node features (BiGCN, EBGCN, RvNN). We notice that the significance levels for the comparisons with WETGN and DUCK$\\_{\\lnot\\text{UT}}$ are relatively lower. This could be attributed to their shared utilization of a transformer backbone and the notable variability in folds. Nonetheless, in comparison to WETGN, which incorporates a top-down GCN with weighted edges, our BiTGN effectively benefits from the bidirectional GCN component, even in the absence of weighted edges. When compared against DUCK$_{\\lnot\\text{UT}}$, which employs two distinct branches of transformers to independently model each conversation thread as both a stream and a graph, our model achieves a more favorable average performance by utilizing only a single transformer branch, thereby resulting in fewer parameters.*”\n7. **Should anonymize the user IDs in social media posts presented in the paper.** \\\nThank you for bringing this to our attention. We recognize the utmost significance of safeguarding users' privacy. To begin with, it's worth noting that all user IDs related to both source posts and responses in the social media content are anonymized in the original paper. Concerning the IDs mentioned within the response content (@'s), we have taken steps to anonymize them as well. The revision will be in the following format: @PolitiFact @joerogan → @name1 @name2.\n\n[1] Jing Ma, Wei Gao, and Kam-Fai Wong. 2018. Rumor detection on Twitter with tree-structured recursive neural networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1980–1989, Melbourne, Australia. Association for Computational Linguistics. \\\n[2] Tian Bian, Xi Xiao, Tingyang Xu, Peilin Zhao, Wen-bing Huang, Yu Rong, and Junzhou Huang. 2020. Rumor detection on social media with bi-directional graph convolutional networks. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 549–556. \\\n[3] Lin Tian, Xiuzhen Zhang, and Jey Han Lau. 2022. DUCK: Rumour detection on social media by modelling user and comment propagation networks. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4939–4949, Seattle, United States. Association for Computational Linguistics. \\\n[4] Tiening Sun, Zhong Qian, Sujun Dong, Peifeng Li, and Qiaoming Zhu. 2022. Rumor detection on social media with graph adversarial contrastive learning. In Proceedings of the ACM Web Conference 2022, pages 2789–2797. \\\n[5] Ruichao Yang, Jing Ma, Hongzhan Lin, and Wei Gao. 2022. A Weakly Supervised Propagation Model for Rumor Verification and Stance Detection with Multiple Instance Learning. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '22). Association for Computing Machinery, New York, NY, USA, 1761–1772. \\\n[6] Fengzhu Zeng and Wei Gao. 2022. Early Rumor Detection Using Neural Hawkes Process with a New Benchmark Dataset. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4105–4117, Seattle, United States. Association for Computational Linguistics. \\\n[7] Landis, J. Richard, and Gary G. Koch. “The Measurement of Observer Agreement for Categorical Data.” Biometrics 33, no. 1 (1977): 159–74. \\\n[8] Shuyang Cao and Lu Wang. 2021. CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6633–6649, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. \\\n[9] Xiuying Chen, Hind Alamro, Mingzhe Li, Shen Gao, Xiangliang Zhang, Dongyan Zhao, and Rui Yan. 2021. Capturing Relations between Scientific Papers: An Abstractive Model for Related Work Section Generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6068–6077, Online. Association for Computational Linguistics. \\\n[10] Khoo, L. M. S., Chieu, H. L., Qian, Z., & Jiang, J. (2020). Interpretable Rumor Detection in Microblogs by Attending to User Interactions. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05), 8783-8790.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ fake news detection is an important research topic\n+ using summarization models in this work is novel and interesting\n+ an adversarial view to fake news detection is valuable", "reasons_to_reject": "- I wonder if it might be possible to look at more recent fake news datasets at *ACL conferences beyond 2019, as the landscape of misinformation could change drastically through time.\n\n- The baseline approaches in this work seem to be solely graph-based approaches analyzing the texts and social media structure. I wonder if non-graph approaches might also be considered as baselines for a fair comparison.\n\n- Section 4.3's title reads \"interpret predictions with summary\", while the section conducts automatic and human evaluation of the quality of generated summaries. I wonder if the authors could better connect summary with interpretability, specifically why the summaries of social media posts provide interpretable evidence of model decision making. Better connecting and highlighting this in the writing is key to motivating the proposed approach.\n\n- Which evaluation metric is adopted in Figure 3? I suggest highlighting that in the caption.\n\n- I suggest providing more information about human evaluation in the appendix. How many annotators are employed? What's the agreement metric among them? What's the compensation scheme? etc.\n\n- I suggest having averaged results over multiple random seeds, variation, and statistical significance tests for Table 4 since the performance gain is sometimes marginal and GNNs could be sensitive to parameter initialization.\n\n- I wonder if it is within ethical research practices to present specific social media posts in the paper with user ids not anonymized? There might be concerns about GDPR rights, etc.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a defend-and-summarize strategy for robust and interpretable fake news detection. Specifically, it leverages abstractive and extractive summarization as interpretability approaches and employs bidirectional graph neural networks for text encoding and classification. Experiments on three datasets demonstrate that the proposed approach outperforms various existing graph-based detection approaches, could alleviate the impact of adversarial responses, and offers interpretation through generated summaries.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: A framework for interpretation of rumors and cospiracy theory has been proposed. Its main principles are focused on the attention on response and to analyze summarization module.

This paper clearly identifies two key drawbacks of existing rumor detectors, lack of robustness against critical responses and lack of interpretability.  Furthermore, the task has been well defined and the experiment results are complete, also providing an interesting ablation study

Some concerns are related to the evaluation of non-graph approaches might also be considered as baselines for a fair comparison, also providing more information about human evaluation in the appendix
Decision: Accept-Main