WSDMS: Debunk Fake News via Weakly Supervised Detection of Misinforming Sentences with Contextualized Social Wisdom
Authors: Ruichao Yang
Wei Gao
Jing Ma
Hongzhan Lin
Zhiwei Yang
Venue: EMNLP 2023 Main (2023)
Keywords: Fake News Detection
Weakly Supervised Learning
Misinformation
Social Network
TL;DR: Debunk fake news via weakly supervised learning by spotting sentence-level misinformation with contextualized social wisdom

Abstract: Fake news debunking primarily focuses on determining the truthfulness of news articles, which oversimplifies the issue as fake news often combines elements of both truth and falsehood. Thus, it becomes crucial to identify specific instances of misinformation within the articles. In this research, we investigate a novel task in the field of fake news debunking, which involves detecting sentence-level misinformation. One of the major challenges in this task is the absence of a training dataset with sentence-level annotations regarding veracity. Inspired by the Multiple Instance Learning (MIL) approach, we propose a model called Weakly Supervised Detection of Misinforming Sentences (WSDMS). This model only requires bag-level labels for training but is capable of inferring both sentence-level misinformation and article-level veracity, aided by relevant social media conversations that are attentively contextualized with news sentences. We evaluate WSDMS on three real-world benchmarks and demonstrate that it outperforms existing state-of-the-art baselines in debunking fake news at both the sentence and article levels.

WSDMS:
Debunk Fake News via Weakly Supervised Detection of Misinforming Sentences with Contextualized Social Wisdom
Ruichao Yang', Wei Gao", Jing Ma'; Hongzhan Lin', Zhiwei Yang'
Honn Kong Baptist University, Hong Kong SAR, China
Singapore Management University, Singapore Jinan University, Guangzhou, Guangdong, China {csrcyang, majing, cshzlin)@ccop. hkbu. edu. hk,
weigaoêsmu. edw.. sg, yangzw@jnu. edu. cn
Abstract
of fake news, thereby complicating the determi- nation of overall veracity (Solovev and Pröllochs, 2022). Such instances have been prevalent during the Covid-19 19 pandemic'.
In recent years, witness the explosion of false and unconfirmed information (i.e., nu- mors) that went viral on social media and shocked the public. Rumors can trigger ver- satile, mostly controversial stance expressions among social media users. Rumor verification and stance detection are different yet relevant lasks. Fake news debunking primarily focuses determining the truthfulness of news arti- cles, which oversimplifies the issue as fake news often combines elements of both truth and falsehood. Thus, it becomes crucial to iden- tify specific instances of misinformation within the articles. In this research, investigate novel task in the field of fake news debunking, which involves detecting sentence-level mis- information. One of the major challenges in this task is the absence of training dataset with sentence-leve annotations regarding ve- racity. Inspired by the Multiple Instance Learn- ing (MIL) approach, we propose a model called Weakly Supervised Detection of Misinforming Sentences (WSDMS). This model only requires bag-level labels for training but is capable in- ferring both sentence-level misinformation and article-level veracity, aided by relevant social media conversations that are attentively con- textualized with news sentences. We evaluate WSDMS on three real-world benchmarks and demonstrate that it outperforms existing state- of-the-art baselines in debunking fake news at both the sentence and article levels.
Fake news detection aims to determine the verac- ity of given news article (Shu et al., 2017). Pre- vious analysis has revealed that users often share comments and provide evidence about fake news on social media platforms (Zubiaga et al., 2017), which has led to growing stream of research that leverages these social engagements, along with the content of news articles, to aid in fake news detec- tion (Pan et al., 201 8; Shu et al., 2019a; Min al., 2022). This approach bears analogies to rumor de- tection, where the focus is on assessing as specific statement rather than an entire news article (Wu et al., 2015; Ma et al., 2018; Bian et al., 2020; Lin et al., 2021; Song et al., 2021; Park et al., 2021; Zheng et al., 2022; Xu et al., 2022). Many studies in this domain aims to train supervised classifiers using features extracted from the social context and the content of the claim article. However, the existing fake news detection models predominately focus on coarse-level classification of the entire article, which oversimplifies the problem. Misin- formation can be strategically embedded within an article by manipulating portions of its content to enhance its credibility (Feng et al., 2012; Rogers et al., 2017; Zhu et al., 2022) Therefore, we target fine- graiggri task that aims identify sentences containing misinformation within article, which can be jointly learned with article-level fake news detection.
1 Introduction
Misinformation, such as fake news, poses tremen- dous risks and threats to contemporary society. The detection of fake news entails various technical challenges (Glockner et al., 2022), and one of them is accurately identifying false elements within news articles. This challenge arises due the blend- ing of authentic and fabricated content by creators
Figure 1 shows an illustrative example of fake news article titled "NASA will pay 100,000 USD
'nttps://ahneddaaadmiirrr wttproooroooouujjjjrroooo com/gujarat- plans- wtw.--e-wwwrr-----wode--e------- to-e-ttobttteee corona/768 77977/6661771111 hhml. This article combines factual information about the historical use of cow urine in India's traditional medicine with false assertions that cow urine contains active ingredients capable of treating Covid-19 19 and has been used in hospitals in South Korea and China.
Jing Ma is the corresponding author.

---

News Title: NASA Will Pay You 100,000 USD Sog in Bed For 60 Doys
5," Worldn' you
cary eleeping Monday moming withoot having
Mondig murming blzes got redyy
5y: type heaves would you envisage if you were paid stay
by. get paid luge stay bed for days,
money just siniing ued for taalell morttss and you know who, NASA no losel!! you the American space agency NASA is paying
$1100
Sx- Most as drean about hangangout bed, all day, every day.
Sq. NASA curuentlyy on U rookoou for people to participate is thak "Bed Rest Studies whith Lugnnnng bugu loal for 60 days stsight.
by It does aound like the drom job, right?
You wouldat jult slegping can keep your self occupted with booka TV, video games and they abo ficir planes they pleasc..
(Tyse'rr to sagr peroo NASS poo for
Past Cheec Doos Niss Study $100 Bal Saly Deen?
Chack Derc Eaxyy
paaty thiy
by Thyy"" could He2 the ronsy
the
Seronalyy Cla you the Lel
coold in
bad
Only 31D2E007 Net coorgh
troough replio
JUNASA bim
Figure 1: A fake news article together with its relevant social context information, where the sentences containing misinformation (i.e., #3 and 85) are orange and posts implying the misinforming sentences are red.
to participants staying in bed for 60 days!", where the sentences in the article can be linked to set of social conversations organized as propagation trees of posts. These sentences contain opinions and evidence that can aid in the veracity classification the sentence and article levels, specifically in spotting misinformation sentences. For instance, sentence 83 can debunked by referring to trees ( and fg, as they provide evidence that contradicts the incorrect reward amount and duration mentioned in the sentence. This information helps in determining that the article is fake. Conversely, if we already know that the article fake, must be misinforming sentences present within it. However, existing methods are not readily appli- cable for the identification of sentence- mis- information due to two main reasons: 1) Obtain- ing veracity labels for sentences in article is costly, it requires annotators to exhaustively fact- check each sentence. 2) While rumor detection models can predict the label of given claim, they often assume the availability of social conversa- tions that correspond to the claim. However, it is difficult to establish a correspondence between so- cial conversations and specific sentences within news article. Inspired by multiple instance learn- ing (MIL) (Foulds and Frank, 2010), attempt develop approach for debunking fake news via weakly supervised detection of misinforming sentences (i.e., instances), called WSDMS2, only using available article-level veracity annotations (i.e., bag-level labels) and a handful of social con- versations related to the news.
methods used in fake news detection that rely on social news engagement data collection (Shu et al., 2020), which provides the necessary conversation trees linked to the article in question. We devise hierarchical embedding model to establish connec- tions between each sentence in the article and its corresponding conversations, facilitating the identi- fication sentence-level misinformation. Standard MIL determines the bag-level label as positive if one or more instances within the bag are positive, and negative otherwise (Dietterich et al., 1997). To improve its tolerance on sentence-level prediction errors, further develop collective attention mechanism for a more accurate article veracity in- ference on top of the sentence-level predictions. The entire framework trained end-to-end by opti- mizing a loss function that aims to alleviate predic- tion bias by considering both sentence- and article- level consistencies. Our approach ensures that the model captures the nuances of misinformation at both levels of granularity. Our contributions are summarized as follows:
infer that there
Unlike existing fake news detection ap- proaches, we introduce new task that is fo- cused spotting misinforming sentences in news articles while simultancously detecting article-level fake news.
We develop WSDMS, biLL-basdd model, to contextualize news sentences aided by social conversations about the news and use only ar- ticle veracity annotations to weakly supervise sentence representation and model training.
To gather the relevant social conversations as- sociated with an article, we employ established
Our method achieves superior performance over state-of ffte-of-ther the-art baselines on sentence- and article-level misinformation detection.
Chttps://gitthee www.HKBUNCCCCECCCC WWWSPMSSSSSSSWWIWWEWSSSSS EMNLP2823

---

2 Related Work
Belis, 2017) to incorporate a weight function over the sentence space calculate the article veracity probability. This assumption allows us to achieve a more robust prediction, as it avoids bias introduced by less important instances.
Early studies on fake news detection have at- tempted to exploit various approaches to extract features from news content and social context in- formation, including linguistic features (Potthast et al., 2018; Azevedo et al., 2021), visual clues (Jin et al., 2016), temporal traits (Kwon al., 2013; Ma al., 2015), user behaviors and profiles (Castillo
Problem Definition
We define a fake news dataset as a of news arti- cles {A]. where each article consists of set of n sentences = {si]]=] and 8; the i-th sentence. For each article, we assume there is a set of m social conversation trees relevant to it denoted as T {tz}}=1 where ty is the j-th conversation tree containing posts (i.e., nodes) and message propaga- tion paths (i.e., edges) which can provide the social context information for A. Our task is to predict the veracity of information at both sentence level and article level in unified model:
al., 2011; Ruchansky al., 2017; Shu et al., 2019b). Subsequent studies have employed neural networks to automatically learn dcep feature repre- sentations from similar sources of data (Ma et al., 2016; Popat et al., 2018; Ma et al., 2019; Nguyen al., 2020; Kaliyar et al., 2021; Sheng et al., 2022). Furthermore, researchers have incorporated exter- nal knowledge sources(Pan et al., 2018; Dun et al., 2021; Hu al., 2021) and combined multi-modal data (Wang et al., 2018, 2021; Fung et al., 2021;
Wu et al., 2021; Silva et al., 2021; Chen et al., 2022) to enhance learning and improve fake news detec- tion performance. Notably, social context informa- tion has played a crucial role in debunking fake news and rumors (Yuan et al., 2019; Khoo et al., 2020; Yang et al., 2022a; Ma et al., 2020; Mehta al., 2022). The utilization of social context struc- tures has spurred the development of Graph Neural Networks (GNNs) such as Kernel Graph Attention Networks (KGAT) (Liu et al., 2020) and Graph- aware Co-Attention Networks (GCAN) (Lu and Li, 2020), which have demonstrated effectiveness in various fake news-related tasks. However, existing approaches (Shu et al., 201 9a; Jin et al., 2022; Yang et al., 2022b) generally aim to detect article-level fake news, which lack the capability to tell which specific sentences contain misinformation.
Sentence-level Veracity Prediction aims to de- termine whether each s; A is misinform- ing sentence or not given its relevant social con- text information T. That to learn a function f(A) 81, ......, 8n - P1. P2, ... "'n where p. is the sentence-level prediction probability as to whether s; is misinforming or not.
Article-level Veracity Prediction aims to clas- sify the veracity of the article A on top of the sentence-level misinformation detection. That is to learn function ((A) P1, P Pn - 1. where ý denotes the prediction to whether A is fake or true. Note that we have only article-level ground truth for model training.
WSDMS: Our MIL-based Model
MIL is a weakly supervised approach that in- fers instance-level labels (e.g., sentence pixel) when training data is annotated with bag-level la- bels (e.g., document or image) (Dietterich al., 1997). Several MIL variants have been developed based on threshold-based MIL assumption (Foulds and Frank, 2010) and weighted collective MIL as- sumption (Pappas and Popescu-Belis, 2017), suc- cessfully applied in various downstream tasks such recommendation systems (Lin et al., 2020) sen- timent analysis (Angelidis and Lapata, 2018), key- words extraction (Wang al., 2016), commu- nity question answering (Chen al., 2017), and more recently joint detection of stances and ru- mors (Yang et al., 2022a). We adopt the weighted collective MIL assumption (Pappas and Popescu-
Detecting more nuanced instances of misinforma- tion at the sentence level solely based on article content is challenging (Feng al., 2012). Previ- ous studies have demonstrated that social media posts contain valuable opinions, conjectures, and evidence that can be leveraged to debunk claim- level misinformation, such as rumors (Ma et al., 2017, 2018; Wu et al., 2019), where claims, typ- ically presented as short sentences, share similar characteristics with sentences in news articles. We hypothesize that the detection of misinforming sen- tences can be done by incorporating relevant in- formation from social context associated with the article. We try to establish connections between social conversations and specific news sentences in the article, enabling the contextualization of social

---

T: Dhnold Truup it drad
SBERT
Syt CNN: Preddent... Asc The world u.. Pans have alraady-
SBERT
SBERT
000
000
Trump's familyio
Article
Transformer
SBER
f
MAX
SBER
00O
000
000
090
Swher
kemel
66O
Conversation trees
Wegghtd collective
Input Embedding
Sentence-Conee Semencc-Coovversation Tree Linking
Misinforming
Article Veracity
Sentence Detection
Prediction
The architecture of our WSDMS model. &, denotes the representation tree t, after kernel-based interaction of post information among tree nodes.
Figure 2:
wisdom to enrich the representation of sentences, in order better capture the veracity of sentences.
followed by model to fix the closeness of each connection; 2) creating link according to the sim- ilarity between s; and tj based on threshold. Our preliminary experiments indicate that the different designs of interaction indeed influence the perfor- mance. Given that the number sentences and trees associated with articles varies significantly, we opt for the threshold- -bsedd approach to avoid the overhead of computing on fully connected graph. We begin with modeling posts interaction in each tree learn its representation before linking the sentences and trees.
The architecture of our MIL-based weakly su- pervised model WSDMS illustrated in Figure 2. WSDMS consists of four closely coupled compo- nents: imput embedding, sentence and conversation tree linking, misinforming sentence detection, and article veracity prediction. We describe them with detail in this section.
4.1 Input Embeddings
We represent the word sequence of each news sentence and social post using SBERT (Reimers and Gurevych, 2019) which maps the sequence into fixed-size vector. Let sequence te W2 liil consist of |.S] tokens, where S could optionally denote a news title, news sentence, or post in conversation tree. Then, the SBERT embedding of can be represented by 5 SBERT(wh,"" ... w)s))) In the rest of the paper. given an article A, we will use additional notations T to denote the news title, and g to denote posts in conversation tree. And then T, 3t, and q will denote the respective SBERT embeddings of T, 8t. p and q.
Post Interaction Embedding: To represent a tree accurately, we use a generic kernel-based graph model KernlGAT (Liu et al., 2020) to mea- sure the importance of each post in a tree by mod- eling the interactions between each post and its neighboring posts.
We first construct translation matrix M to rep- resent the similarity of each post with its neighbors, where each Ma E .M is the cosine similarity be- tween post and q:
P.
if (
M pm
(1)
otherwise
where (p)) is the set of neighboring nodes of p. We then define kernel function (MM,) to rep- resent the features considering the interactions be- tween p and its neighbors based on K Gaussian kernels (Keerthi and Lin, 2003), and this yields:
4.2 Linking Sentences to Conversation Trees
To mine the discernible relationship between sen- tences and social posts trees, we want to design sentence-tree tree linking mechanism between the sen- tence set [2 and post tree set (4))7 both associated with A. There are clearly different de- signs to create links across the elements between them, such as 1) using a fully connected graph that links any 8; and ty regardless of their similarity,
G(M,) (G1 (x(M))
(2)
where
(Myy 1x)
(M(M) log Σ exp
20
gEN(p)

---

and Pa and o% are parameters in the k-th kernel capture the node interactions different lev- els (Xiong et al., 2017). Note that if 0% -> oo, the kernel function degenerates to the mean pooling.
tion. This is achieved by:
expll, 82)
Big Egjer, expll', (.0()'),))) (8))
(4)
Then, update the representation ý of each post p by considering all its neighbors with their identified importance, which is given as:
Σ Bog GCT
8,
where 8; denotes the socially contextualized rep- resentation of 8,, Pig is the importance of tj E T; with respect to S;. and @ denotes concatenation operation.
(Wi (wi (G(M)) + bi) [a]
7/4 softmax
Σ (CN(p)
(3)
p- =
7q
We then fully-connected softmax layer predict the probability of s; containing misinfor- mation based on its BERT- embedding 8, and socially contextualized embedding 8;;
where "/ is scalar representing the post-level at- tention coefficient between p and its neighbor q. Wi and bj are trainable parameters used to trans- form K kernels into a vector of all nodes in the tree, [q] takes the value corresponding to post and 인 and q are initialized respectively with the BERT-based post embeddings and T.
p. = softmar + W33, + b2)
(5)
where W2, W3 and by are trainable parameters and Pi is the class probability distribution of 8; pro- vided that the bag-level class labels are fake and real, based on the MIL (Foulds and Frank, 2010; Angelidis and Lapata, 2018).
Link Sentences and Trees. With the obtained interaction- eenannce post representations, we use a mean pooling function to represent a conversation tree fj. i.e., Ij = mean(,, p) for all E 1j. For each pair of sentences and tree (st, f2) associated with an article, then create link between them if the cosine similarity of 8; and ły is above global threshold 7, where 7 determined according the global range of similarity scores between sentences and trees by mapping 7 to the median of the range of scores. We fix this setting empirically.
4.4 Inferring Article Veracity
We can simply predict an article as fake if there is at least one misinforming sentence is detected, which conforms the original threshold-basee MIL assumption. However, the assumption is overly strong because there can be inaccuracies in sentence-level prediction. Based on the weighted collective MIL assumptions (Foulds and Frank, 2010), we design a context-based attention mecha- nism to bridge the inconsistency between sentence- and article-level predictions.
4.3 Detecting Misinforming Sentences
To spot misinforming sentences based the graph with the sentence-tree links, we propose a graph attention model to detect whether a sentence 8; contains misinformation. Each sentence can be linked to multiple conversation trees and vice versa. In Figure 1, for example, two trees l and l3 are linked to $3. where & provides more specific evi- dence (c.g., the right reward amount and the num- ber of experimental days) indicating that $3 is mis- informing, while la just implies the sentence is not credible without providing specific clues. Hence, we design an attention mechanism to update the representation of each sentence by considering the importance of all its corresponding trees.
Specifically, we first learn global represen- tation for the article utilizing a pre-trained trans- former (Grail al., 2021):
[T, $1.... $,] Trans (1 ..,.,,))))) (6)
where T the initial SBERT embedding of the article title. We then adopt an attention mechanism to measure the importance of sentences w.r. the article veracity prediction, which yields:
exp(s Q))
Σ exp(8..
9
a4 Ps
More specifically, let T denote the of trees linked to 8;- We aggregate the representation of corresponding trees according to their attention weights, and then update the sentence representa-
1=1
where O₁ denotes the attention weight of & rela- tive to the title representation T, and ğ is the class probability distribution of A being fake or real.

---

4.5 Model Training
Stat.
PolitiFact
GossipCop
BuzzNews
Intuitively, the more similar two sentences are, the more similar their corresponding predictions should be. We define the following loss function considering pairwise consistency between sentence representation and prediction, with only article- level ground truth:
Train
# True
624
16,658
301
Fake
432
5,255
# True
140
160
50
Test
# Fake
70
80
25
Total
1,270
22,153
481
# avg. sent/art # uvg. trees/art
30
27
27
13
16
9
avg. posts/tree
316
340
C(A) = A C(A)+ (1- - 118A - BAll2
(8)
Table 1: Statistics of the datasets used.
where
5.2 Article-level Fake News Detection
We compare the following models at the article level. Some original settings of baselines might not suit the data in this task, which have to be specifically customized (see Appendix A.I). 1) DeClarE (Popat al., 2018): An evidence-aware network using news title to attend over words in rel- evant posts for verifying news claims. 2) HAN (Ma et al., 2019): A hierarchical attention network us- ing the news title to attend over relevant posts as evidences. 3) dEFEND (Shu et al., 2019a): A sentence- post co-attention network for fake news detection. 4) BerTweet (Nguyen al., 2020): A language model pre-trained 850M tweets, which is applied here for article verification using article and relevant posts. 5) GCAN (Lu and Li, 2020): graph-aware co-attention model trained on user profile and post propagation structure without us- ing post content to verify the news given title. 6) Bi-GCN (Bian et al., 2020): A bi-directional graph convolutional network using news title and prop- agation structure of posts for verifying the news. 7) KAN (Dun al., 2021): An attention network utilizing entities in article content and entity con- texts for fake news detection. 8) SureFact (Yang et al., 2022b): A reinforcement subgraph reasoning method using the topic connection between article and relevant posts for fake news detection. 9) WS- DMS: Our proposed weakly supervised method. 10) WSDMS-FC: A variant of our method that fully connects sentences and post trees. Table 2 presents the following observations:
C(A)
exp (-118; - 8,112 l1 2)113)
j=1
Here C(.) с [0, 1] is the function measuring the consistency between pairwise sentence similarity (i.e., 8; and 8,) and the prediction (i.e., fi; and (z)) M4 and AA denote respectively the ground-trun trunthhhtuhhhee and predicted class probability distributions of A. 11-113 is an efficient kernel based on the L2 norm (Luo al., 2016) as non-negative penalty function, and À is the trade-off coefficient.
5 Experiments and Results
5.1 Datasets and Setup
We employ two public real-world datasets Politi- Fact and GossipCop (Shu et al., 2020) respectively related to politics and entertainment fake news, where relevant social conversations are collected from Twitter. We also construct an open-domain fake news dataset BuzzNews by extending Buz- zFeed (Tandoc Jr, 2018), for which gather social conversations of the articles via Twitter AP13.
We recruit three annotators to label misinform- ing sentences of the articles in the test sets of the three datasets. We train the annotators by providing them with unified set of annotation rules referring the detailed guide from several fact-checking websites such snopes.com and politifact.com. where specific rationales on how each claim was judged are provided. Then, we take a majority vote for determining the label of each sentence, and the inter-annotator agreement is 0.793. Table shows the statistics of these three datasets.
the first group structured models, dEFEND performs the best. This is because DeClearE and HAN are designed to only use the external rele- vant context of a claim and BerTweet is trained represent social posts. dEFEND leverages features extracted from both article content and external posts that are complementary.
We precision (Pre), recall (Rec), F1, and accuracy (Acc) as evaluation metrics. All the base- lines and our methods are implemented with Py- Torch (Paszke et al., 2019) (see Appendix A.2 for implementation details).
"https://developer: twi wwiiterrccmm. com/en//occs

---

Dataset
PolitiFact
CossipCop
BuzzNews
Method
Pre
Rec
F1
Acc
Pre
Rec
Fl
Acc
Pre
Rec
F1
Acc
DeClarE
0.714
0.746
0.730
0.789
0.706
0.741
0.723
0.762
0.705
0.743
0.724
0.754
HAN
0.752
0.779
0.765
0,803
0.718
0.739
0.728
0.789
0.727
0.768
0.747
0.762
dEFEND
0.900
0.926
0.913
0,886
0.729
0.785
0.756
0,808
0.731
0.792
0.760
0,810
BerTweet
0.844
0.903
0.873
0.878
0.851
0.862
0.857
0.848
0.831
0.840
0.835
0.811
GCAN
0.817
0.821
0.819
0.837
0.782
0,803
0.792
0.791
0.780
0,800
0.790
0.795
Bi-GCN
0.852
0.838
0.845
0,865
0.797
0.813
0.805
0.822
0.791
0.814
0.802
0.817
KAN
0.870
0.840
0.855
0,859
0.776
0.770
0.773
0.807
0.766
0.790
0.778
0,820
SureFact
0.913
0.939
0.924
0.887
0.859
0.872
0.865
0.847
0.841
0.856
0.848
0,829
WSDMS
0.921
0.967
0.943
0,904
0.864
0.876
0.870
0.850
0.850
0.857
0.853
0.858
WSDMS-FC
0.923
0.967
0.944
0.908
0.862
0.879
0.871
0.853
0.850
0.857
0.853
0.858
Table Article-level fake news detection results.
In the second group of non-structured models, the graph-based models GCAN and Bi-GCN mainly rely on propagation structures of fake news and perform comparably with KAN using entities and their contexts extracted from the social me- dia content, suggesting that social conversations embed good amount of human wisdom use- ful for detecting fake news. SureFact performs best among all the baselines because it groups social posts into the topics discovered from arti- cle content, suggesting that creating a connection between them at the topic level is helpful.
train all models on the same training data. But this intentionally disadvantages our WSDMS since it can only use article labels. Therefore, we also present the performance of WSDMS (o) trained on the original training sets without sentence labels, which baselines cannot take advantage of. Table 3 conveys the following findings:
Similar to article-level prediction, dEFEND out- performs DeClarE and HAN because effec- tively models the sentence and social context correlations via the co-attention mechanism. BERTweet is more advantageous at representing social media posts, demonstrating better perfor- mance at the sentence level.
WSDMS consistently defeats the best baseline SureFact on the three datasets, demonstrating that our explicit and fine-grained linking between sentence and social context is superior, and the sentence-level detection can help article verac- ity prediction. In addition, WSDMS does not sacrifice its performance compared to WSDMS- FC that uses full connections between sentences and trees, while we find that WSDMS signifi- cantly reduces training time from 4.5 to 2 hours. This indicates our sentence-tree linking method is cost-effective.
Among the structured models, KAN performs best because it incorporates both content and propagation information and has a co-attention mechanism between sentence and entity con- texts extracted from social conversations. This may enhance sentence representation better than Bi-GCN and GCAN that can only utilize propagation-busse features.
Weakly supervised WSDMS performs better than DeClarE and comparably with HAN, which are fully supervised. This is because WSDMS con- siders the propagation structure while DeClarE and HAN can only leverage unstructured posts. The overall performance of WSDMS is clearly compromised due to weak supervision. How- ever, when it is trained on the original datasets, WSDMS (o) can enjoy the large volume of ar- ticle labels to beat all baselines that cannot be weakly supervised. To reach the same level of performance, the baselines may need tremendous sentence annotations which are infeasible to get. Again, it performs comparably well as WSDMS- FC (o), implying that our sentence-tree linking
5.3 Misinforming Sentence Detection
For misinforming sentence detection, the base- lines are deployed by treating each sentence as a claim and the conversation trees linked to the sen- tence (see Section 4.2) as the source of evidence. SureFact is excluded as it cannot classify specific sentences. More details are in Appendix A.
Since all baselines are supervised methods that need sentence labels for training, we split the three test sets with sentence-level annotation into train and test parts with a 70%-30% ratio. Due to the large number of sentences in the original test sets (6.30006,48022,480), we end up with three work- able sentence-level training and test sets. We then

---

Dataset
PolitiFact
GossipCop
BuzzNews
Method
Pre
Rec
F1
Acc
Pre
Rec
F1
Acc
Pre
Rec
F1
Acc
DeClarE
0,504
0.531
0.517
0.559
0.501
0.528
0.514
0.550
0.513
0.520
0,516
0.540
HAN
0.531
0.559
0.545
0.565
0.510
0.529
0.519
0.561
0.518
0.537
0.527
0.562
dEFEND
0.539
0.586
0.562
0.605
0.534
0.581
0.557
0.600
0.538
0.570
0.554
0.580
BerTweet
0,542
0.630
0.583
0.619
0.539
0.619
0.576
0.602
0.542
0.610
0.574
0.599
GCAN
0.533
0.563
0.548
0.589
0.511
0,561
0.535
0.581
0.521
0.551
0.536
0.580
Bi-GCN
0,557
0.589
0.573
0.606
0.531
0,560
0.545
0.593
0.533
0,553
0,543
0.601
KAN
0.574
0.594
0.584
0.611
0.539
0.561
0.550
0.609
0.540
0.560
0.550
0.610
WSDMS
0.518
0.539
0.527
0.564
0.508
0.531
0.519
0.562
0.513
0.537
0.524
0,549
WSDMS (o)
0.637
0.676
0.655
0.644
0.629
0.664
0.646
0.639
0.609
0.587
0.598
0.662
WSDMS-FC (o)
0.639
0.679
0.658
0.650
0.633
0.664
0.648
0.639
0.610
0.590
0.600
0.662
Table 3: Minformaing sentence detection results.
reserves vital information for spotting misinform- ing sentences efficiently.
Sentence-leve
Article-tevel
0.65
0.95
WITMIG
WSDMS effectively enhances sentence-level performance by utiliizigg publicly accessible article-level labels. To achieve comparable per- formance, baseline systems generally require massive fine-grained sentence-level annotations. Consequently, sentence-level prediction remains pivotal contribution of our study.
0.60
0.90
NL.
TTee as sart
Aur lonttel
Wid Enea
0.80
Acc
Figure 3: Ablation results PolitiFact dataset.
5.5 Case Study
To gain deeper insight, we visualize two news articles checked by PolitiFact in Figure 4 which predicted fake (left) and true (right) correctly by WSDMS. The spotted misinforming and true sentences are also shown. We observe that 1) WS- DMS can associate a sentence with multiple trees using attention weights (arrow lines indicate high- weight trees) to help determine its veracity. 2) The posts in the conversations provide useful clues for indicating how credible each sentence is by aggre- gating collective opinions of users in the trees; 3) The article-level veracity is not determined sim- ply by whether there is a misinforming sentence detected, because the prediction might be inaccu- rate. For example, if $4 incorrectly predicted as fake, the article will also be determined as fake un- der the standard MIL. Our approach increases the chance of correcting such an error by giving higher attention weights to other sentences, which may indicate that the article is overall more likely to be true. Thus, the attention weights of sentences can collectively aggregate sentence-level predictions to improve the final prediction.
5.4 Ablation Study
We ablate WSDMS based on the PolitiFact dataset by varying some component(s): 1) w/o т: Fully connect sentences and trees by removing т, i.e., WSDMS-FC. 2) w/o NLL: Replace the loss with an ordinary negative fog-likelihood loss function. 3) w/o we: Infer article veracity based on the orig- inal MIL assumption without weighted collective attention. 4) Title as sent: Treat the title com- mon sentence. 5) w/o kernel: Reduce the kernel- based post interaction embedding to dot-product attention between sentence and conversation trees. 6) w/o tree: Remove conversation trees.
Figure 3 shows that most of the ablations make the result worse. w/o tree implies that only us- ing article content insufficient for the task. w/o kernel supports that embedding post interactions with kernel can help post and tree representation. Experiment in the Appendix A.3 also echoes the advantages of the kernel. Title as sent means that the news title may attract the most attention from the trees, which can hurt the representation of other sentences, and should be specially treated. w/o indicates adopting weighted collective MIL is bet- ter. w/o NLL confirms that our designed loss is necessary and effective. Only w/o T is marginally better due to fully connected sentences and trees, which is however more costly and less efficient.
5.6 User Study Experiment
We conduct a user study to evaluate the quality of the model output. We sample 120 articles from PolitiFact and present them in two forms: Baseline (article, posts) and WSDMS (article, misinforming

---

Donald Trump dead from a farel
Thlle: Hillury Chinton vays puin
these various forms of misinformation. Whereas, we simplify some techniques in this paper. For ex- ample, the representation of conversation trees can be learned by considering the direction of message propagation and combining top-down and bottom- up propagation trees. In addition, it cannot deal with more complex situations, where multiple true sentences combined constitute some kind of logical falsehoods inconsistencies. This can be strength- ened by considering sentence-level relations such as discourse information in the model. Despite this limitation, WSDMS encounters no such situ- ation in the three datasets used according to our observation. Nevertheless, this suggests that the ex- isting fake news datasets and detection models lack consideration of discourse-level fakes or logically inconsistent compositions, which presumably not uncommon in real-world fake news, Lastly, we only use social context data collected from Twitter, which might have platform bias. To mitigate the is- sue, we can introduce additional data from different social media platforms, such as BuzzFace (Santia and Williams, 2018) from Facebook.
lendiny conme doath for yourey bordd
Donald Trump was pronounedd
"4: Gim viclence by for lealing icune....
Bimous besinesrtion and
rV.
In speech posted media and
Fant have already taken social
wi Far We collected data of common catises...
Trauph mnd my
True
Truw Now loman
dlss woalle woodd botlee des
Mitgpeeee to marrgingg
Ievere wite
Figure 4: case study illustrating the prediction.
F1
Acc
Confidence
Avg. Time/news
Baseline
0.784
0.795
2.017
10 see
WSDMS
0.958
0.989
3.206
sec
Table 4: User study results on model outputs quality.
sentences, trees). We then ask 6 users to label the articles and give their confidence in a 5-point Likert Scale (Joshi et al., 2015), and each person is given only one form to avoid cross influence.
Table 4 shows that 1) users determine the article- level veracity more accurately with WSDMS: 2) users spent 70% less time identifying fake news; and 3) users show higher confidence with the re- sults of WSDMS, suggesting that users tend to be more sure about their decision when specific misinforming sentences and relevant evidence are provided.
Acknowledgements
This work was partially supported by Hong Kong RGC ECS (Ref. 22200722), National Natural Science Foundation of China Young Scientists Fund(No. 62206233) and HKBU Onc-off Tier 2 Start-up Grant (Ref. RCOFSGT220022/SCTO0))
6 Conclusion and Future Work
Ethics Statement
We propose a MIL-based model called WSDMS to debunk fake news in a finer-grained manner via weakly supervised detection of misinforming sen- tences with only article veracity labels for model training. WSDMS uses the attention mechanism to associate news sentences with their relevant so- cial news conversations identify misinforming sentences and determine the article's veracity by aggregating sentence-level predictions. WSDMS outperforms a set strong baselines at the article level and sentence level on three datasets.
Data Privacy: Although the datasets used in our re- search are publicly accessible, the utilization of so- cial media conversations for debunking fake news may raise concerns regarding user privacy. To ad- dress this issue, we took measures to anonymize all social media posts during the data processing and experiments, ensuring that user information remains invisible and unusable. Additionally, our proposed approach does not require access to any sensitive user information, therefore climinating the risk of privacy infringement. The collection of social media conversations in the BuzzNews dataset was conducted in compliance with the pri- vacy regulations set by the platform.
In the future, we will incorporate more inter- sentence features, such as discourse relations, to detect composition-level misinformation.
Social Implications: The detection and debunking of fake news can carry significant social and politi- cal implications. One critical consideration is the potential impact on the reliability of the system and the possibility of misleading users by mislabeling
Limitations
Fake news is one type of misinformation, which also includes disinformation, rumors, and propa- ganda. WSDMS can be generazzed to detect

---

information misinformation vice versa. In light of this concern, we have taken precautions to carefully assess the model we developed and restrict their distribution to the general public. We are committed to designing a responsible policy regarding the dissemination of codes and datasets within research community, and ensure that they are used responsibly in manner that aligns with ethical standards and societal well-being.
In Proceedings of the eleventh ACM international conference on web search and data mining, pages 126-134.
Thomas G Dietterich, Richard H Lathrop, and Tomás Lozano-Pérez. 1997. Solving the multiple instance problem with axis-parallel rectangles. Artificial intelligence, (911-)331-7
Yaqian Dun, Kefei Tu, Chen Chen, Chunyan Hou, and Xiaojie Yuan. 2021. Kan: Knowledge-aware atten- tion network for fake news detection. In AAAL
Song Feng, Ritwik Banerjee, and Yejin Choi. 2012. Syntactic stylometry for deception detection. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 171-175.
References
Stefanos Angelidis and Mirella Lapata. 2018. Multi- ple instance learning networks for fine-grained sen- timent analysis. Transactions of the Association for Computational Linguistics, 6:17-31.
James Foulds and Eibe Frank. 2010. multi-instance learning assumptions. The knowledge engineering review, 25(1):1-25.
review of
Lucas Azevedo, Mathieu d' Aquin, Brian Davis, and Manel Zarrouk. 2021. Lux (linguistic aspects under examination): Discourse analysis for automatic fake classification. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 41-56.
Yi Fung, Christopher Thomas, Revanth Gangi Reddy, Sandeep Polisetty, Heng Ji. Shih-Fu Chang. Kath- leen McKeown, Mohit Bansal, and Avirup Sil. 2021. Infosurgeon: Cross-media fine-grained infor- mation consistency checking for fake news detec- tion. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume Long Papers), pages 1683-1698.
Tian Bian, Xi Xiao, Tingyang Xu, Peilin Zhao, Wen- bing Huang, Yu Rong. and Junzhou Huang. 2020. Rumor detection on social media with bi-directional graph convolutional networks. In Proceedings of the AAAI conference on artificial intelligence, vol- ume 34, pages 549-556.
Max Glockner. Yufang Hou, and Iryna Gurevych. 2022. Missing counter-evidenee renders nlp fact-checking unrealistic for misinformation. In Proceedings of the 2022 Conference on Empirical Methods in Natural
Carlos Castillo, Marcelo Mendoza, and Barbara Poblete. 2011. Information credibility on twitter. In Proceedings of the 20th international conference on World wide web, pages 675-684.
Language Processing, pages 5916-5936.
Quentin Grail, Julien Perez, and Eric Gaussier, 2021. Globalizing best- transformer architectures for long document summarization. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1792 1810.
Yixuan Chen, Dongsheng Li, Peng Zhang, Jie Sui, Qin Lv, Lu Tun, and Li Shang. 2022. Cross-modal am- biguity learning for multimodal fake news detection. In Proceedings of the ACM Web Conference 2022, pages 2897-2905.
Zheqian Chen, Ben Gao, Huimin Zhang, Zhou Zhao, Haifeng Liu, and Deng Cai. 2017. User personal- ized satisfaction prediction via multiple instance deep learning. In Proceedings of the 26th International Conference World Wide Web, pages 907-915.
Linmei Hu, Tianchi Yang, Lubao Zhang, Wanjun Zhong, Duyu Tang, Chuan Shi, Nan Duan, and Ming Zhou. 2021. Compare the knowledge: Graph neural fake news detection with external knowl- edge. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 763.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019. What does bert look at? an analysis of bert's attention. ACL 2019, page 276.
Yiqiao Jin, Xiting Wang, Ruichao Yang, Yizhou Sun, Wei Wang. Hao Liao, and Xing Xie. 2022. Towards fine-grained reasoning for fake news detection. Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 7754-
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of machine learning research, 12ARTICLLE)299325577
Zhiwei Jin, Juan Cao, Yongdong Zhang, Jianshe Zhou, and Qi Tian. 2016. Novel visual and statistical im- age features for microblogs news verification. IEEE transactions on multimedia, 19(3):598-608.
Zhuyun Dai, Chenyan Xiong, Jamie Callan, and Zhiyuan Liu. 2018. Convolutional neural net- works for soft-matching n-grams in ad-hoc search.

---

Ankur Joshi, Saket Kale, Satish Chandel, and Kumar Pal. 2015. Likert scale: Explored and explained. British journal of applied science & technology, 7(4):396.
Association for Computational Linguisties, pages 2561-2571.
Jing Ma, Wei Gao. Shafiq Joty, and Kam-Fai Wong. 2020. An ceantton-bbsed rumor detection model with tree-structured recursive neural networks. ACM Transactions Intelligent Systems and Technology (TIST), 11(4):1-28.
Rohit Kumar Kaliyar, Anurag Goswami, and Pratik Narang. 2021. Fakebert: Fake news detection in so- cial media with bert-hased deep learning approach. Multimedia Tools and Applications, 80(8): 1765- 11788.
Jing Ma, Wei Gao, Prasenjit Mitra, Sejeong Kwon, Bernard Jansen, Kam-Fai Wong, and Meeyoung Cha. 2016. Detecting rumors from microblogs with recurrent neural networks.
S Sathiya Keerthi and Chih-Jen Lin. 2003. Asymptotic behaviors of support vector machines with gaussian kernel. Neural computation, 15(7): (177):6677-189)
Jing Ma. Wei Gao, Zhongyu Wei, Yueming Lu. and Kam-Fai Wong. 2015. Detect rumors using time series of social context information on mi- croblogging websites. In Proceedings of the 24th ACM international conference information and knowledge management, pages 1751-1754.
Ling Min Serena Khoo, Hai Leong Chieu, Zhong Qian. and Jing Jiang. 2020. Interpretable rumor detec- tion in microblogs by attending to user interactions. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 8783-8790.
Jing Ma. Wei Gao, and Kam-Fai Wong. 2017. De- tect rumors in microblog posts using propaga- tion structure via kernel learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pages 708-717.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: In
A method for stochastic optimization. 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Sejeong Kwon, Meeyoung Cha, Kyomin Jung, Wei Chen, and Yajun Wang. 2013. Prominent features of rumor propagation in online social media. In 2013 IEEE 13th international conference on data mining, pages 1103-1108. IEEE.
Jing Ma, Wei Gao. and Kam-Fai Wong. 2018. Rumor detection on twitter with tree-structured recursive neural networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1980- 1989.
Hongzhan Lin, Jing Ma, Mingfei Cheng, Zhiwei Yang. Liangliang Chen, and Guang Chen. 2021. Rumor detection on twitter with claim-guided hierarchical graph attention networks. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10035-10047
Nikhil Mehta, María Leonor Pacheco, and Dan Gold- wasser. 2022. Tackling fake news detection by continually improving social context representa- tions using graph neural networks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1380.
Yusan Lin, Maryam Moosaci, and Hao Yang. 2020. Out- fitnet: Fashion outfit recommendation with attention- based multiple instance learning. In Proceedings of The Web Conference 2020, pages 77-87.
Erxue Min, Yu Rong, Yatao Bian, Tingyang Xu, Peilin Zhao, Junzhou Huang, and Sophia Ananiadou. 2022. Divide and ann-cunquer: Post-user interaction network for fake news detection on social media. In Proceedings of the ACM Web Conference 2022, pages 1148-1155 1158.
Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. 2020. Fine-grained fact verification with kernel graph attention network. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7342-7351.
Dat Quoc Nguyen, Thanh Vu, and Anh-Tuan Nguyen. 2020. Bertweet: A pre-trained language model for english tweets. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing: System Demonstrations. pages 94.
Yi-Ju Lu and Cheng-Te Li. 2020. Gean: Graph- aware co-attention networks for explainable fake news detection social media. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 505-514.
Jeff Z Pan, Siyana Pavlova, Chenxi Li, Ningxi Li, Yangmei Li. and Jinshuo Liu. 2018. Content based fake news detection using knowledge graphs. 17th International Semantic Web Conference, ISWC 2018, pages 669-683. Springer Verlag.
Xiong Luo. Xiaohui Chang, and Xiaojuan Ban. 2016. Regression and classification using extreme learning machine based on 11 -orm and 12-norm. Neurocomputing. 174(PA): 17--186.
Jing Ma, Wei Gao, Shafiq Joty, and Kam-Fai Wong. 2019. Sentence-level evidence embedding for claim verification with hierarchical attention networks. In Proceedings of the 57th Annual Meeting of the
Nikolaos Pappas and Andrei Popescu-Belic 2017. Ex- plicit document modeling through weighted multiple- instance learning. Journal of Artificial Intelligence Research, 58:591-626.

---

Sungkyu Park, Jamie Yejean Park, Hyojin Chin, Jeong- han Kang, and Meeyoung Cha. 2021. An experi- mental study to understand user experience and per- ception bias occurred by fact-checking messages. In Proceedings of the Web Conference 2021, pages 2769-2786
Kai Shu, Deepak Mahudeswaran. Suhang Wang, Dong- won Lee, and Huan Liu. 2020. Fakenewsnet: A data repository with news content, social context, and spa- tiotemporal information for studying fake news on social media. Big data, 8(3):111-188
Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. 2017. Fake news detection on social me- dia: A data mining perspective. ACM SIGKDD explorations newsletter, 19(1):22-36.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style. perf-peeformance deep learning library. Advances neural information processing systems, 32.
Kai Shu, Xinyi Zhou, Subang Wang, Reza Zafarani. and Huan Liu. 2019b. The role of user profiles for fake news detection. In Proceedings IEEE/ACM international conference on advances in social networks analysis and mining, pages 436-439.
the 2019
Kashyap Popat, Subhabrata Mukherjee, Andrew Yates, and Gerhard Weikum. 2018. Declare: Debunk- ing fake news and false claims using evidence- aware deep learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 22-32.
Amila Silva, Ling Luo, Shanika Karunasekera, and Christopher Leckie. 2021. Embracing domain dif- ferences in fake news: Cross-domain fake news tection using multi-moda data. In Proceedings of the AAAI conference on artificial intelligence, vol- ume 35, pages 557-563.
Martin Potthast, Johannes Kiesel, Kevin Reinartz, Janek Bevendorff, and Benno Stein. 2018. A stylomet- ric inquiry into hyperpartisan and fake news. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume
Kirill Solovev and Nicolas Pröllochs. 2022. Moral emo- tions shape the virality of covid- misinformation on social media. In Proceedings of the ACM web conference 2022, pages 3706-3717.
1: Long Papers), pages 231-240.
Nils Reimers and Tryna Gurevych. 2019. Sentence- hert: Sentence emheddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IICNLP), pages 3982 3992.
Yun-Zhu Song. Yi-Syuan Chen, Yi-Ting Chang, Shao-Yu Weng, and Hong-Han Shuai. 2021. Adversary-aware rumor detection. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1371-1382.
Todd Rogers, Richard Zeckhauser, Francesca Gino, Michael Norton, and Maurice Schweitzer. 2017. Artful paltering: The risks and rewards of using truthful statements mislead others. Journal of
Edson C Tandoc Jr. 2018. Five ways buzzfeed is preserving (or transforming) the journalistic field. Journalism, 19(2):20--216.
personality and social psychology, 112(3):456.
Wei Wang, Yue Ning, Huzefa Rangwala, and Naren Ramakrishnan. 2016. A multiple instance learn- ing framework for identifying key sentences and detecting events. In Proceedings of the 25th ACM International Conference on Information and Knowledge Management, pages 509-518.
Natali Ruchansky, Sungyong Seo, and Yan Liu. 2017. Csi: A hybrid deep model for fake news detection. In Proceedings of the 2017 ACM Conference on Information and Knowledge Management, pages 797-806.
Yaqing Wang, Fenglong Ma, Zhiwei Jin, Ye Yuan, Guangxu Xun, Kishlay Jha, Lu Su, and Jing Gao. 2018. Eann: Event adversarial neural networks for multi-modal fake news detection. In Proceedings the 24th acm sigkdd international conference on knowledge discovery & data mining, pages 849-857.
Giovanni C Santia and Jake Ryland Williams. 2018 Buzzface: A news veracity dataset with facebook user commentary and egos. In Twelfth international AAAI conference on web and social media.
Qiang Sheng, Juan Cao, Xueyao Zhang, Rundong Li, Danding Wang, and Yongchun Zhu. 2022. Zoom and observe: News environment per- ception for fake news detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pages 4543-4556.
Yaqing Wang, Fenglong Ma, Haoyu Wang, Kishlay Jha, and Jing Gao. 2021. Multimodal emergent fake news detection via meta neural process networks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 3708 3716-
Kai Shu, Limeng Cui, Suhang Wang, Dongwon Lee, and Huan Liu. 2019a. defend: Explainable fake news de- tection. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery data mining, pages 395-405.
Ke Wu, Song Yang, and Kenny Q Zhu. 2015. False rumors detection on sina weibo by propagation struc- tures. In 2015 IEEE 31st international conference on data engineering, pages 651-662. IEEE

---

Liang Wu, Fred Morstatter. Kathleen M Carley, and Huan Liu. 2019. Misinformation in social me-
A
Appendix
dia: definition, manipulation, and detection. ACM SIGKDD Explorations Newsletter, 21(2):80-90.
A.1 Detailed Baseline Settings
Existing fake news detection and rumor detection methods predominately focus on coarse-level clas- sification on the entire article and claim, respec- tively, while our goals include identifying misin- forming sentences within an article at a fine-grained level. When comparing with the baselines that are originally designed to either classify a news article claim, the required (and available) in- puts may differ from our study. Therefore, we need to specifically customize the data inputs make the baselines applicable to the article-level and sentence-level detection tasks while maintain- ing the implementation of baseline models intact. In this section, we will provide more details about baseline models and the information they used.
Yang Wu, Pengwei Zhan, Yunjian Zhang. Liming Wang, and Zhen Xu. 2021. Multimodal fusion with co-attention networks for fake news detection. In Findings of the association for computational linguistics: ACL-IJCNLP 2021, pages 2569.
Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. 2017. End-to-end neural ad-hoc ranking with kernel pooling. In Proceedings of the 40th International ACM SIGIR conference on research and development in information retrieval, pages 55-64.
Weizhi Xu, Junfei Wu, Qiang Liu, Shu Wu, and Liang Wang. 2022. Evidence-aware fake news detection with graph neural networks. In Proceedings of the ACM Web Conference 2022, pages 2501-2510.
A.1.1 Article-level Task
Ruichao Yang, Jing Ma, Hongzhan Lin, and Wei Gao. 2022a. A weakly supervised propagation model for rumor verification and stance detection with mul- tiple instance learning. In SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid. Spain, July 11 15, 2022, pages 1761-1772. ACM.
1) DeClarE (Popat et al., 2018) is designed to clas- sify claim with relevant news content obtained from external sources as evidence, such as web search results. The claim it used is short and there many relevant articles providing evidence. In our fake news detection dataset, however, what is available includes single long-form article which is the target to be checked, and the relevant social conversation trees providing external assistance. Since DeClarE can only accept short claims as in- put, we use the title of the news article as an input claim and the posts in conversations evidence. 2) HAN (Ma al., 2019) aims similarly to De- ClarE to the claim verification task and the pro- vided evidence set collected from multiple docu- ments relevant to the claim. In our case, article text is the target to be verified, while HAN assumes a short claim as the target which cannot be fed into HAN directly. So, we use the news title the input claim and posts in conversations evidence.
Ruichao Yang, Xiting Wang, Yiqiao Jin, Chaozhuo Li, Jianxun Lian, and Xing Xie. 2022b. Reinforce- ment subgraph reasoning for fake news detection. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2253-2262.
Chunyuan Yuan, Qianwen Ma, Wei Zhou, Jizhong Han, and Songlin Hu. 2019. Jointly embedding the local and global relations of heterogemeous graph for rumor detection. In 2019 IEEE international conference on data mining (ICDM), pages 796- 805, IEEE
Jiaqi Zheng, Xi Zhang, Sanchuan Guo, Quan Wang, Wenyu Zang, and Yongdong Zhang. 2022. Mfan: Mudti-modal feature-enhanced attention networks for numor detection. IJCAL.
3) dEFEND (Shu al., 2019a) is fake news detection model using news article as the target of verification and the related user comments as evidence. This is mostly consistent with our setting. Thus, it does not require any special treatment. 4) BerTweel (Nguyen et al., 2020) is a pre- trained language model trained on large English posts corpus. It is designed to encode short text. To apply BerTweet for article-level verification, we use the posts in conversation trees to fine-tune the model, and then treat the news title as a claim to be verified because BerTweet cannot accept the
Yongchun Zhu, Qiang Sheng, Juan Cao, Shuokai Li, Danding Wang, and Fuzhen Zhuang. 2022. Gen- eralizing to the future: Mitigating entity bias in fake news detection. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2120-2125.
Arkaitz Zubiaga, Alex Voss, Rob Procter, Maria Liakata, Bo Wang, and Adam Tsakalidis. 2017. Towards real- time, country-level location classification of world- wide tweets. IEEE Transactions on Knowledge and Data Engineering, 29(9):2053-2066

---

Method
Article-level
Sentence-level
DeClarE
news content. Posts as evidence.
Sentence as claim. Linked posts as evidence. Sentence claim. Linked posts evidence. Senlence as news content. Linked posts as evidence.
HAN
Title content. Posts evidence.
dEFEND
Article as news content. Posts evidence.
BerTweet
Title as news content. Posts fine-tune model.
Sentence as claim. Linked posts fine-tune the model.
GCAN
Title news content. Users of posts as evidence.
Senlence as claim. Users of linked posts as evidence. Sentence as claim. Linked posts as evidence. Sentence as claim. Entities from sentence and linked posts.
BI-GON
Title as news content. Posts as evidence
KAN
Article as news content. Entities from articles and posts.
SureFact
Title
content. Posts evidence.
Table 5: Application of baselines to suit the fake datasets while keeping their original implementation intact.
fong-form article as input.
A.2 Implementation Details
5) GCAN (Lu and Li, 2020) aims at debunking rumors only using the corresponding sequence of retweet users without text comments of a source tweet. The source tweet it accepts as claim is also short. To apply to our data, we use the news title as source tweet and the post user profiles and propagation structure without post content as evidence.
Our model parameters are updated by back- propagation (Collobert et al., 2011) with Adam (Kingma and Ba, 2015) optimizer. We set the maximum epoch 100, the dimension of embeddings 512 for sentences and posts, and empirically initialize the learning rate as 0.001, and the hyperparameter validated on a small hold-out dataset. As for Gaussian kernels in Equation 2, we set K = 10. Here one kernel with parameter 44 = and σы 0.001 is designed for exact match- ing (Dai al., 2018). The other kernels' parameter T% 0.01, and their parameter μr is distributed within [-1, 1] evenly.
set to 0.5 which is
6) Bi-GCN (Bian et al., 2020) utilizes bi- directional Graph Convolutional Network to ac- commodate top-down and bottom-up post prop- agation structure to detect rumors taking a short source post as input. Similarly, we use news title source post and post propagation structure as evidence.
The training process controlled to end when the loss value converges or the maximum epoch number is met.
7) KAN (Dun et al., 2021) detects fake news by identifying entity mentions in news contents and align them with the entities in the knowledge graph, which are used to learn news-entity co-attentions for better representing news text. While there news articles in our data, have only related posts from social media but no knowledge graph. For this issue, we use the social conversions of the article as the source to extract entities as entity contexts the entities in the article.
A.3 Experiment on Kernel Attention Concentration
We conduct experiment to compute the entropy values of kernel attention weights used in WSDMS and compare it with dot-product attention used in GCAN, reflect whether the learned attention weights are more focused or scattered. The lower the entropy, the more focused the attention mecha- nism (Clark et al., 2019). The entropy results are given in Table 6.
8) SureFact (Yang al., 2022b) groups related posts based on specific topics extracted from news content implicitly connect news and social media content for fake news detection. It can be directly applied to our datasets.
kernel
dot-product
Attention Entropy
5.11
6.03
A.1.2 Sentence-level Task
Table 6: Entropy score of kernel attention and dot- product attention.
For misinforming sentence detection, the baselines are deployed by treating sentence in article as claim source post and the conversation trees linked to the sentence (see Section 4.2) the source of evidence. In such a setting, most of the baselines can be applied to this sentence-level task in a more straightforward manner. See Table 5 for specific details.
We find that kernel attention bears a smaller en- tropy than the dot-product attention. It suggests that kernel attention has a stronger ability to be focused on a few more vital posts. This is also the reason why we use kernel attention in our method.


Reviews:
[{"review_id": "ClamwobIz2", "reviewer": "Reviewer_BSaA", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your great effort and positive feedback on our submission! \n\n1.\tIt is true that the BERT base model can be replaced with other state-of-the-art representation methods such as Sentence-BERT. The main reason we just used BERT is to have a fairer comparison with the baselines such as BerTweet, KAN and SureFact, all of which employed BERT for sentence representations. We ensure to include Sentence-BERT-based experimental result into our revised paper. \n\n2.\tWe agree that fake news detection is a hard problem requiring more challenging dataset. To the best of our knowledge, the three datasets we used are representative and widely known. Besides, the datasets have already considered imbalanced classes where the number of fake instances is around 30-50% of the number of true instances, as shown in Table 1. We conjecture that the datasets creators could further reduce the ratio to make the fake class even sparser, but the difficulty might be how to know what the realistic (or gold) ratio is in real-world situations. It is a very interesting research question that we are keen to find out in the future, and we’re not aware of any existing fake news datasets representing or approximating such a gold ratio of fake vs. true. If you know any references regarding the real-world fake news ratio or any more realistic dataset, we’ll be very thankful and determined to test out our approach on that.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents a model to address a challenging problem, detecting sentence-level misinformation and article-level veracity, and evaluates the performance of the proposed model on three real-world benchmarks. The literature is critically presented, and the authors considered social conversations for sentence-level predictions. The overall contribution of the study is shown by the depth of the experiments.\n\n", "reasons_to_reject": "This paper, while intriguing and worthy of consideration for publication, employs certain outdated methods like the base BERT model for sentence representations, which may not be optimal compared to models like Sentence-BERT. The chosen datasets for experimentation are widely known and relatively straightforward; testing on more challenging datasets, where fake news is sparse as in real-world situations, would provide a more genuine insight into its applicability. Since fake news detection is a hard problem, so complex datasets should be used to address this task. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a model called Weakly Supervised Detection of Misinforming Sentences (WSDMS) to detect sentence-level misinformation and article-level veracity and evaluates the performance of the proposed model on three real-world benchmarks. Drawing from existing literature, the well-articulated research showcases results that eclipse the state-of-the-art (SOA) benchmarks. The model adeptly predicts sentence veracity and, combined with attention weights, guides the article's overall truthfulness assessment, proving instrumental in enhancing fake news detection while underscoring the significance of sentence-level prediction. \n\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oo8CJcJp6P", "reviewer": "Reviewer_HG47", "summary": "", "questions": "- Why did you chose BERT over other methods that could be more effective in modelling sentence-level representations?\n- In your paper you use the term misinformation a bit as an umbrella term. In the literature, there is a small but relevant difference in meaning between misinformation and disinformation, pertaining the intentionality (i.e., misinformation is mostly unintentional, while disinformation is intentional). I understand the use of the term in your context, but I believe that it could be worth it to clarify this aspect in the paper. ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful feedback and suggestions! Essentially, these are very similar concerns to the Reviewer BSaA.\n\n1.\tAs we explained above, we used BERT to have a fairer comparison with the baselines, such as BerTweet, KAN, and SureFact which also used base BERT. We are committed to including experimental results using Sentence-BERT in our revised paper. \n\n2.\tWe agree that fake news detection requires more challenging dataset. The datasets we used present certain degree of class imbalance as shown in Table 1, where the number of fake news accounts for 30-50% of that of true news. We concur that this might not a reflection of the real-world ratio, but the difficulty is we do not know what the real-world ratio is, and we’re not aware of any existing dataset reflecting a real-world ratio. We think it is a very interesting research question to answer in the future. Again, if you know any references regarding the real-world fake news ratio or any more realistic dataset, we’ll be very thankful and determined to test out our approach on that! \n\n3.\tThanks again for your helpful suggestion on the use of umbrella term “misinformation”. Indeed, we didn’t differentiate misinformation and disinformation strictly here even though we’re aware of their difference. We will clarify it and define disinformation to be a subset of misinformation that our approach can generally deal with in the revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper faces a challenging problem in a creative but also effective way. While the approach leverages many existing techniques, this is not a shortcoming, as authors are able to incorporate them in their own method.\nWhile not a definitive answer to the problem of misinformation, I believe that it may be an interesting stepping stone of the branch of the literature focusing on social contexts.", "reasons_to_reject": "The paper leverages few seemingly outdated or out-of-scope approaches, e.g. a base BERT model for sentence-level representations, which is far from optimal. Maybe a different model like Sentence-BERT would have performed better for the same computational cost. \nAlso, the experiments are conducted on three well know and generally easy datasets. In my opinion, experiments on more challenging datasets (e.g., those were fake news are orders of magnitude less than real ones, as in real-world scenarios) are better suited for actually improving real-world fake news detection.", "questions_to_authors": "- Why did you chose BERT over other methods that could be more effective in modelling sentence-level representations?\n- In your paper you use the term misinformation a bit as an umbrella term. In the literature, there is a small but relevant difference in meaning between misinformation and disinformation, pertaining the intentionality (i.e., misinformation is mostly unintentional, while disinformation is intentional). I understand the use of the term in your context, but I believe that it could be worth it to clarify this aspect in the paper. ", "paper_topic_and_main_contributions": "The paper provides an approach to sentence-level and article-level misinformation detection using weak supervision.\nThe authors leverage many findings of existing literature to devise their strategy, which is appreciated. \nThe paper is clear, well written, and present results that surpass SOA on the provided datasets.\nThe architectural and experimental section appear to be sound.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper is well written. My only minor gripe is the one mentioned above concerning the term misinformation, that I think could be changed or clarified to reflect the intentions of the authors.\n"}, "extra_text": {}}, {"review_id": "BPMKlohXeq", "reviewer": "Reviewer_np1j", "summary": "", "questions": "Question A: Why is there no emphasis on fact-checking? The user study is also never mentioned in the main body. In my opinion, this would be a strong direction for future research.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful review and feedback! \n\n1.\tW1 – Sentence-level-only Performance. Let us clarify the rationale behind the proposed WSDMS model: No existing fake news detection dataset provides labels of misinforming sentences, while all sentence-level baseline models require sentence labels for training. Given the tremendous costs of locating and annotating sentence-level misinformation in news articles, it is impractical to scale up such sentence-labeled datasets (e.g., to the size of the original training dataset), rendering the baseline models unable to get fully trained. WSDMS inherently is not bound by this limitation as it doesn’t require sentence-level annotation to predict sentence labels. Therefore, it can enjoy a much larger training set of new articles (with article-level annotations only) to boost its sentence-level prediction performance like what WSDMS (o) has demonstrated, while the baselines models can’t do this at all with the same training data. We believe this is a unique advantage of our model and it is fair to demonstrate the advantage in this way. \n\n\n\tBasically, we want to convey that WSDMS can boost sentence-level performance by training on abundantly available article-level labels, but the baselines can’t. For baselines to get comparable performance, there is an impractical high bar to prepare fine-grained sentence-level annotations. Therefore, sentence prediction is one of our main contributions. \n\n\n\tAs for the WSDMS trained on the smaller test-training data with sentence annotations, which didn’t show an advantage compared with baselines, it is not surprising since its capability is intentionally weakened in such setting given a very limited number of article labels available (147/168/53). We think this doesn’t detriment the strength of WSDMS. If predominant emphasis of sentence prediction in the title and abstract still makes people feel uncomfortable, we will adjust our tone and add some specific discussions following the text in Section 5.3. \n\n2.\tW2 - Reproducibility: We will release the dataset publicly. For the source code, we will consider sharing it more responsibly. Rather than directly posting codes on a public webpage, we plan to only post the detailed usage and performance together with a request form for people who are interested in our code to log their contact and explain why they need the code and for what purpose. Then, we will verify each request via direct communication with the requesters to ensure that it will be used for research purposes only, which will be liable for a legal agreement. We believe this will be compatible with our ethical statement and do not harm the reproducibility of our method by other researchers.\n\n3.\tQuestion A - No emphasis on fact-checking: Fact-checking is a related area. We commented it out after we quickly balanced the space limit and the fact that our problem setting, datasets, and the angle of approaching to the misinformation challenge are different. Typically, fact-checking aims to verify a given claim rather than a long-form news article. Also, our method needs to locate and predict the misinforming sentences within the article which is also not considered in existing fact-checking tasks. We will add the related content back in the revised paper. For the user study, we moved it to the appendix completely while forgetting to leave a mention of it in the main text. We will put it back to the main text and discuss its value for future research. Since one more page of content will be allocated in the camera-ready paper, we will definitely have enough space to contain these discussions if accepted.\n\n4.\tWe will correct the typos and add the reference papers accordingly to the revised paper. Thanks!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "S1 - Approach: Considering social conversations for sentence-level predictions is interesting, feasibly conducted, and with promising results.\nS2 - Experiments: The range of experiments (including choice of datasets and baselines) logically follows the overall contribution.\nS3 - Case/User Study: The case and the Appendix-only user study also demonstrate the benefits of the model. Moreover, I encourage the authors to include a brief discussion of the user study in the main paper.", "reasons_to_reject": "W1 - Sentence-level-only Performance: The sentence-level performance evaluation in Table 3 is (as noted by the authors) unfair. WSDMS is mediocre due to its disadvantage in the task (no sentence labels). WSDMS (o), on the other hand, has an advantage (pre-trained on train set). Hence, I find it troublesome to emphasize sentence prediction as prominently in the title and abstract.\nW2 - Reproducibility: If possible, I expect releasing the novel datasets and models. Also, see my comment in the ethical section.", "questions_to_authors": "Question A: Why is there no emphasis on fact-checking? The user study is also never mentioned in the main body. In my opinion, this would be a strong direction for future research.", "paper_topic_and_main_contributions": "The paper introduces a multiple-instance-learning-based WSDMS model, which (similar to rumor detection) uses social conversations to support fake news detection. WSDMS predicts sentence veracity, which (together with attention weights) informs the article veracity prediction. The experiments show that the proposed approach leads to improved fake news prediction at the article level, while demonstrating the benefits of the sentence-level prediction task. The contribution is timely and relevant for EMNLP.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "027-036: The motivation of the problem is weak and lacks references to support the claims. The paper only lists a single article as an example. \n\nBelow, I suggest two papers that tie in neatly to strengthen the motivation. The first provides evidence that COVID-19 misinformation has increased virality. The second highlights the challenges of automatic fact-checking. \n\nSolovev, K., & Pröllochs, N. (2022, April). Moral emotions shape the virality of COVID-19 misinformation on social media. In Proceedings of the ACM web conference 2022 (pp. 3706-3717).\nGlockner, M., Hou, Y., & Gurevych, I. (2022). Missing counter-evidence renders nlp fact-checking unrealistic for misinformation. arXiv preprint arXiv:2210.13865.", "Typos_Grammar_Style_and_Presentation_Improvements": "Inconsistencies:\n- threads vs. trees\n- 076: Referring to t_2 is confusing here, as Figure 1 only points to t_1 and t_3.\n- 238:-243: different naming than Figure 2.\n\nTypos:\n- 379: s_i should be s^hat_i?\n- Figure 2: WSDM (missing an S).\n\nOther writing issues:\n- 345: Briefly clarify non-standard symbols like ⊕. Here, it likely refers to a direct sum.\n- 414: Provide more details on the guide used by the annotators.\n- Table 2, 457, 464: Briefly explain how groups 1 and 2 differ. Non-structured vs. structured?", "Justification_for_Ethical_Concerns": "While not requiring an ethics review, I find the statement about not publicly releasing the model due to ethical reasons misplaced (650-656). Such restrictive practices harm reproducibility and trust in the system. Additionally, an open model would enable checking the reliability and verifying (mis-)labeling. Hence, the authors seem to act opposite to their stated ethical concerns. "}}]

Author rebuttal:

Meta-review: This paper introduces the task of identifying sentence-level misinformation, as part of fake-news debunking. Due to the lack of an appropriate dataset, multi-instance learning is used as a form of weak supervision, and achieves state of the art results.

Overall, reviewers (all of whom were quite confident) were Impressed by both the writing ("well-articulated research") and the research ("faces a challenging problem in a creative but also effective way").

Two reviewers raised nearly identical concerns -- the use of an outdated model as the basis for the method, when a newer approach (SentenceBERT) might have performed better, and the use of datasets that are known to be relatively easy. The authors have committed to including results with the first suggestion in their final version, but note that they are limited by the lack of a more suitable dataset for the second.

Although these do seem like shortcomings, reviewers were nearly unanimous in this paper being strong on both soundness and excitement. As such, this seems like a clear accept ("the contribution is timely and relevant"). None were particularly effusive in their written praise (e.g., "intriguing", and an "interesting stepping stone", rather than transformative), but all rated it as strong on excitement, and wish to see it published.
Decision: Accept-Main