Identifying Conspiracy Theories News based on Event Relation Graph
Authors: Yuanyuan Lei
Ruihong Huang
Venue: EMNLP 2023 Findings (2023)
Keywords: conspiracy theory
misinformation
event relation graph

Abstract: Conspiracy theories, as a type of misinformation, are narratives that explains an event or situation in an irrational or malicious manner. While most previous work examined conspiracy theory in social media short texts, limited attention was put on such misinformation in long news documents. In this paper, we aim to identify whether a news article contains conspiracy theories. We observe that a conspiracy story can be made up by mixing uncorrelated events together, or by presenting an unusual distribution of relations between events. Achieving a contextualized understanding of events in a story is essential for detecting conspiracy theories. Thus, we propose to incorporate an event relation graph for each article, in which events are nodes, and four common types of event relations, coreference, temporal, causal, and subevent relations, are considered as edges. Then, we integrate the event relation graph into conspiracy theory identification in two ways: an event-aware language model is developed to augment the basic language model with the knowledge of events and event relations via soft labels; further, a heterogeneous graph attention network is designed to derive a graph embedding based on hard labels. Experiments on a large benchmark dataset show that our approach based on event relation graph improves both precision and recall of conspiracy theory identification, and generalizes well for new unseen media sources.

Identifying Conspiracy Theories News based on Event Relation Graph
Yuanyuan Lei and Ruihong Huang Department of Computer Science and Engineering Texas A&M University, College Station, TX (yuanyuan, huangrh)@tamu.cr edu
Abstract
are widespread throughout the world (van Prooijen and Douglas, 2018). was estimated that more than half of the US population believed in at Ieast one conspiracy theory (Oliver and Wood, 2014). The widespread presence of conspiracy theories can cause harm to both individuals and society whole (Van der Linden, 2015), such as reduc- ing science acceptance, introducing polarization, driving violence, obstructing justice, and bringing public health risks (Hughes et al., 2022; Leonard and Philippe. 2021) etc. Thus, developing novel models to detect conspiracy theories becomes im- portant and necessary.
Conspiracy theories, type of misinforma- tion, are narratives that explains an event or situation in irrational or malicious manner. While most previous work examined conspir- acy theory in social media short texts, limited attention put such misinformation in long news documents. In this paper, we aim identify whether news article contains con- spiracy theories. We observe that conspiracy story can be made up by mixing uncorrelated events together, by presenting an unusual dis- tribution of relations between events. Achiev- ing a contextualized understanding of events in story essential for detecting conspiracy theories. Thus, we propose to incorporate an event relation graph for each article, which events are nodes, and four common types of event relations, coreference, temporal, causal, and subevent relations, are considered as edges. Then, integrate the event relation graph into conspiracy theory identification in two ways: event-aware language model is developed to augment the basic language model with the knowledge of events and event relations via soft labels; further, heterogeneous graph at- tention network is designed to derive graph embedding based on hard labels. Experiments large benchmark dataset show that our ap- proach based on event rela relation graph improves both precision and recall of conspiracy theory identification, and generalizes well for new un- seen media sources!.
Most previous work studies conspiracy theories primarily from psychology or communication per- spectives, often concentrating on social media short texts (De Coninck et al., 2021; Mari et al.. 2022). Few work focused developing computational models to detect conspiracy theories, especially for lengthy news articles. However, substantial portion conspiracy theories originate from long documents on news websites (Miani et al., 2021), and then disseminate virally through various social media channels (Cinelli et al., 2022; Moffitt et al., 2021). Therefore, we focus on identifying con- spiracy thcories within these long documents from news websites, and aim to comprehend the underly- ing narratives. This is challenging task which not only requires semantic understanding (Tangherlini et al., 2020) but also necessitates logical reasoning (Georgiou et al., 2021).
1 Introduction
One observation we have for identifying conspir- acy theories is that a conspiracy story can be made up by mixing unrelated events together. The con- cept of "event" refers to specific occurrence action that happens around us, and is essential in story telling and narrative understanding (Zhang et al., 2021). Conspiracy theories, however, usually put typically unrelated events together to fabricate a false story. Take the article in Figure as an example, the author includes both "Murder" event
Conspiracy theories are narratives that attempt to explain the significant social or political events as being secretly plotted by malicious groups at the expense of an unwitting populations (Douglas et al., 2019; Katyal, 2002). A variety of conspiracy theories ranging from science-related moon land- ing to political-related pizzagate (Bleakley, 2021)
'The code and data link: http://gituh.ccoyyyaaauaaal nip/conspiracy thoories_ catp__mmnpp000p 2023

---

Murder: most coronavirus patients died duc treatment ventilators... recent study, researchers tracked people infected with covid-19, and 320 patients required hospitalization, However, 88 pereentage of them died while being treated on ventilators... It's time to stop the ventilator treatment!
died
trealment
Murder
coreference
corsference
died
treated
overlup
confforence relation
subevent
tomporal relation
infected
hospitalization
cunsal relation
before
An example of conspiracy theory news article, and its corresponding event relation graph. Events are shown in bold text. Event relation graph consists of events as nodes, and four types of event-event relations as links. conspiracy story can be made up by mixing unrelated events (nodes) together, or by presenting an irrational conspiracy logic between events (event relations).
Figure I:
Conspiracy
Benign
Singleton
Temporal
Causal
Suhevent
Singleton
Temporal
Causal
Subevent
84.61 %
30.67%
24.18%
11.91 %
76.81%
43.25%
10.68%
4.71%
Table I: Percentage of singleton events, temporal events, causal events, and subevents all events reported conspiracy theory news and benign news. The higher ratio values shown in bold. Conspiracy theory articles involve less in coreference and temporal relation, but involve more in causal and subevent relation.
and "treatment ventilators" event in the arti- cle, trying to persuade the readers that treatment on ventilators is equivalent to murdering, which is questionable and immediately against our intuition. To effectively guide the model to recognize the irra- tionality of mixing unrelated events like "Murder" and "treatment on ventilators" together, maybe by drawing upon rich world knowledge already coded in underlying language model, propose encode events information for conspiracy theo- ries identification.
(before, after overlap) between events, causal causality between events, and subevent contain- ment from a parent event to a child event.
A statistical analysis of events involving the four types of event relations is shown in Table 1, based the LOCO conspiracy theories dataset (Miani et al., 2021). The numerical analysis confirms the following observations:
(1) Conspiracy theories news involves less in coreference relations, imply- ing that unlike a normal news that reports events centered around the main event, a conspiracy the- ories news tends to be more dispersed in story telling. (2) Conspiracy theories exhibit less tem- poral relations, which means that events reported in conspiracy theories news adhere less well to the chronological narrative structure than those events presented in mainstream benign news. (3) Conspir- acy theories news contains more causal relations and shows a tendency to ascribe more causality to certain events, thereby illustrating reasons or potential outcomes. (4) Conspiracy theories news employs subevent relations more frequently, elabo- rates more details, provides verbose explanations, and tends incorporate more circumlocution.
Furthermore, observe that conspiracy the- ories often rely on presenting relations between events in an unusual way. Revisiting the example in Figure 1, the author intends to convince readers through an irrational logical chain of events: people were infected with covid-19 and required hospital- ization, but died while being treated on ventilators, therefore it is the ventilator treatment that led to their death. However, the normal rational logic is that people's 's death just co-occurred with treatment the same time, it was not really the treatment that caused these deaths. The conspiracy theories here presents an invalidate and baseless causal re- lation by emphasizing the co-occurring temporal relation. To sensitize the model to unusual and il- logical event-event relations, we propose to equip the model with event relations knowledge. In par- ticular, we incorporate four common types of event relations that are crucial for narrative understanding and logical reasoning (Wang et al., 2022): corefer- ence whether two event mentions designate the same occurrence, temporal chronological orders
Motivated by the above observations and anal- ysis, we propose to incorporate both events and event-event relations into conspiracy theories iden- tification. More specifically, an event relation graph is constructed for each article, in which events are nodes, and the four types of relations between events are links. This event relation graph explic- itly represents content structures of story, and guides the conspiracy theories detector to engage

---

its attention on significant events and event-event relations. Moreover, this event relation graph is incorporated in two ways. Firstly, an event-aware language model is trained using the soft labels de- rived from the event relation graph, thereby inte- grating the knowledge of events and event-event relations into the language model. Secondly, het- erogeneous graph attention network is designed to encode the event relation graph based on hard la- bels, and derive a graph feature vector embedded with both events and event relations semantics for conspiracy theories identification. Experiments on the benchmark dataset LOCO (Miani et al., 2021) show that our approach based on event relation graph improves both precision and recall of con- spiracy theory identification, and generalizes well for new unseen media sources. The ablation study demonstrates the synergy between soft labels and hard labels derived from the event relation graph. Our contributions are summarized as follows:
models (Ma et al., 2016; Jin et al.. 2017) were designed to extract semantic features (Truică and Apostol, 2022). Beyond lexical and semantic anal- ysis, our work aims to comprehend the irrational logic behind conspiracy theories, from the perspec- tive of events and event relations.
Fake News Detection is to verify whether the news article is fake or real (Pérez-Rosas al., 2018; Ru- bin et al., 2015; Shu et al., 2017; Hassan et al., 2017; Thorne and Vlachos, 2018). Although both fake news and conspiracy theories involve misinfor- mation, fake news detection focuses on checking the existence of false information, while conspir- acy theories tend to explain a happened event in a malicious way through irrational logic (Avramov et al., 2020). In this paper, attention is concen- trated on exploring the intricate narratives and the underlying logic of conspiracy theories.
Event and Event Relations have been studied for decades. An event refers to an action or occur- rence in the real world (Wang et al., 2020). The four types of event relations we consider, corefer- ence, temporal, causal and subevent relations, are all commonly seen event-event relations in stories. However, the four types of event relations were previously studied in isolation(Zeng et al., 2020; Bethard et al., 2012; Tan et al., 2022; Yao et al., 2020). Instead, we consider all the four types of event relations to depict event-level discourse struc- tures and achieve contextualized understanding of events in narrative. Our approach is enabled by the recently introduced large dataset MAVEN-ERE (Wang et al., 2022) that has the four event relations annotated within individual articles.
To the best of our knowledge, our model is the first computational model for detecting conspiracy theories news articles.
identify events and event relations as crucial information for conspiracy theories detection.
design a new framework to incorporate event relation graph for conspiracy theories detec- tion.
2 Related Work
Conspiracy Theory research till now mainly stud- ied short comments from social media such as Twit- ter (Wood, 2018; Phillips et al., 2022), Facebook (Smith and Graham, 2019), Reddit (Klein et al., 2019; Holur et al., 2022), or online discussions (Samory and Mitra, 2018; Min, 2021; Mari et al., 2022). However, large amount of conspiracy the- ories are sourced from long narratives on news websites and shared through the url link in social media platforms (Moffitt et al., 2021). Therefore, we aim to develop the model to identify conspiracy theories in these news articles.
3 Methodology
In this section, we explain the details of event rela- tion graph construction for each article. The event relation graph incorporated within two steps. Firstly, am event-aware language model is devel- oped based on the soft labels, with events and event relations knowledge augmented. Secondly, het- erogeneous graph neural network is designed to derive a graph embedding based on the hard labels. Figure 2 illustrates our proposed methodology.
Misinformation Detection was studied for years, such as rumor (Meel and Vishwakarma, 2020), pro- paganda (Da San Martino et al., 2019), political bias (Fan al., 2019; Lei et al.. 2022; Baly al., 2020; Lei and Huang, 2022). Farly efforts utilized lexical analysis (Wood and Douglas, 2013). With the advent of deep learning, neural network-based based
3.1 Event Relation Graph Construction
We need identify events as well as extract the four types of event relations to create an event re- lation graph for each individual article. We will describe the training process and the graph con-

---

event relation graph construction
soft label
eveut-aware language model
event .veevent 2
congfrcusc
semporal
hand label
conral
comspiracy benign
aubevent
doc node
document
event identification
relutions
event relation graph encoder
Figure 2: An illustration of conspiracy theory news identification based on event relation graph
struction process separately.
fied, the next step to extract and classify the four types of relations. Given a pair predicted events (event; eventý), the trained coreference identifier predicts the probability for coreference relation as:
For training, we use the annotations from the general-domain MAVEN-ERE dataset (Wang et al., 2022). First, we train the event identifier, where the Longformer (Beltagy et al., 2020) language model is used to encode the entire article and classifica- tion head is built top of the word embeddings predict whether the word triggers event or not. Next, we train four event relation extractors. Following the previous work (Zhang et al., 2021; Yao et al., 2020), we form the training event pairs in the natural textual order, where the former event in the pair is the precedent event mentioned in text. Regarding temporal relations we process the be- fore annotation as follows: keep the label as before if the annotated event pair aligns with the natural textual order, or assign after label to the reverse pair if not. The simultaneous, overlap, begins-on, ends-on, contains annotations are grouped into the overlap category in our event relation graph. Re- garding causal relations, we assign the causes label if the natural textual order is followed, and assign caused by label otherwise. Similarly for subevent relations, assign the contains label if the natural textual order is followed, and assign contained by label otherwise. Since the events and four event relations interact with each other to form a cohesive narrative representation, we adopt the joint learning framework from (Wang et al., 2022) to train these components collaboratively.
peorefer now honn-ooree corefer corefery
(Pi.
PP.)
corofer is the probability of i-th and j-th where Pij
events corefer with each other. We also utilize the trained temporal classifier generate the predicted probability for temporal relation:
P (pig .PC. .Po) .Pi
non-temp before ootter queriap)
non-temy denotes the probability no tem- where Pij
poral relation between the i-th and j-th events, and before after ocerlop represents the probability Pij .Pi. .Pij of before, after, overlap relations respectively. Sim- ilarly, the trained causal classifier and subevent classifier generate the predicted probability
ponnaal non non--onee cousal gusce casceed by)
(p.
Pig
(4)
poubevent
Pi (p2
PiJ
contasn contained-by
P...
(5)
Finally, the predicted probabilities in eq(1)-(5) are incorporated as the soft labels for events and event relations in the constructed event relation graph. Accordingly, the hard label for each element is obtained by applying the argmaz function to the predicted probabilities.
During the event relation graph construction process, the initial step is event identification. Given candidate news article that consists of N words, the trained event identifier generates the predicted probability for each word:
3.2 Event-aware Language Model
To incorporate the constructed event relation graph into conspiracy theories identification, we first leverage the soft labels to train an event-aware lan- guage model, with the knowledge of events and event relations augmented. The soft labels provide fuzzy probabilistic features and enable the basic language model to be aware that nodes represent events and links represent event relations.
pevent
(p) Lpnoo-eee event .P.
evuntt
(1)
where = I,.......... N. and puuent is the probability of i-th word being an event. With the events identi-
'Time expressions such as date or time are also annotated in temporal relations. However, our event relation graph fo- cuses on events, so we exclude temporal annotations related to time expressions and solcly retain annotations hetween events.
Considering the news articles are typically long, we utilize Longformer as the basic language model

---

(Beltagy al., 2020). In order to capture the con- textual information, we add an extra layer of Bi- LSTM on top (Huang et al., 2015). Given a candi- date news article, the embedding of each word is represented as (10, W02,-.,, .-..
knowledge within the event relation graph can be augmented into the basic language model:
Loss, = Σ Pl, log(Q(,)
(12)
WNN)
To integrate events knowledge into the basic language model, we construct two-layer neural network on top of the word embeddings to learn the probability of each word triggering an event:
where r E {corefer, temp, causal, subevent represents the four event relations.
The overall loss for training the event- lan- guage model based soft labels is computed the sum of the losses for learning each component:
Qu (g. cone--et event ) (12tmaar(WaWWree + bi) + ba)
(6)
Los8soft =1.088eeett + Lo8Scorefer + L088temp (13)
where 1,.... N,, and Qrvent is the learned prob- ability of the basic language model. The soft la- bel pewent generated by the event relation graph is referenced as the target learning materials. By minimizing the cross entropy loss between the learned probability Queent and the target proba- bility prnent the events knowledge from the event relation graph can be infused into the basic lan- guage model:
Lo88rausal + Lo88
3.3 Event Relation Graph Encoder
We further design heterogeneous graph neural net- work to encode the event relation graph using hard labels. The encoder updates events embeddings with their neighbor events emheddings through in- terconnected relations, and produces a final graph embedding to represent each news article. This final article embedding is encoded with both events and event relations features, and will be later uti- lized for conspiracy theories identification.
N pevent log(Q
Los
To capture the global information of each doe- ument and explicitly indicate the connection be- tween each document and its reported events, we introduce an extra document node and connect it to the associated events, as shown in Figure 2. The document node is initialized as the embedding of the article start token 
, and the event node is initialized as the event word embedding.
To integrate the event relations knowledge into the basic language model, we build four different neural networks on top of the event pair embedding. to learn the predicted probability of the four event relations respectively:
Qurefer (9 non corffer comeeaor)
R.
(8)
soʃ Iimaammaaaaa (Wana(Wa(Walee.. @ eq) + bs) + bu)
The resulting event relation graph comprises nine fine-grained heterogeneous relations: coref- erence, before, after, overlap, causes, caused by, contains, contained by relations constructed from hard labels, as well as the event-doc relation. The eight event-event relations inherently carry seman- tic meaning, whereas the event-doc relation is standard link without semanties. In order incor- porate the semantic meaning of event relations into graph propagation, we introduce the relation- aware graph attention network. In terms of the event-doc relation, we utilize the standard graph attention network (Veličković et al., 2018).
Q
=
(0.)) Hi.
non temptemp before after (pverlap)
9i3 .1..
softmar(Wo(Waa e @e,) bb) bu)
Q Ccassll
non consal cause byy
9h
(10)
so softmar(Ws e;) br) ha)
rodevent (g subevent
contain I,, Vig
contained- ba)
1 e;) 3))+ + by) + b1o)
(11)
where (es + Ej) is the embedding for event pair (eventa, eventy) by concatenating the two events embeddings together. Quarefer Qlempp Q Quausal and Q Quybevent are the learned probability of the ba- sie language model for the four event relations. The soft labels generated by the event relation graph Pig feorefer Plemp Pegusel, paubevent contain rich event relations information, and thus are referenced as the target learning materials. By minimizing the cross entropy loss between the learned probabil- ity and the target probability, the event relations
The relation-aware graph attention network is designed to handle event-event relations, with the relations semantics integrated into event nodes em- beddings. Given a pair of events (eventy, event,). their relation rty is initialized as the embedding of the corresponding relation word. At the I-th layer, the input for i-th event node are output features

---

produced by the previous layer denoted as ng 1) During the propagation process, the relation em- bedding between i-th and j-th event is updated as:
train
dev
test
conspiracy
39
11
8
benign
43
27
22
= w" (1 ri ① h "1
Table Number of media sources in the train dev test set in the media source splitting setting
(14)
where B represents feature concatenation and W is trainable matrix. Then the attention weights aij across neighbor event nodes are computed as:
train
dev
test
conspiracy 4729
1421
1581
benign
14321
5028
5095
Oaj = sofbmar, (W* (15)
Table 3: Number of articles in the train / dev / test set in the media source splitting and random splitting settings
The output features h are formulated as:
10
Σ JENL.
aiy anyWVro réj
(16)
4 Experiments
4.1
Dataset
where W W WV are trainable matrices, and Nr, denotes the neighbor event nodes connecting with i-th event via the relation type r. After collect- ing h for all relation types E R |coreference, before, after, overlap, causes, caused by, contains, contained by), aggregate the final output feature for i-th event 1-th layer as:
Most prior work studied conspiracy theories within social media short text. LOCO (Miani et al., 2021) is the only publicly available dataset that provides conspiracy theories labels for long news documents. LOCO consists of large number of documents col- lected from conspiracy theories media sources and 92 mainstream media sources. LOCO labels articles from conspiracy theories websites with the conspiracy class, and articles from mainstream me- dia sources with the benign class. To prevent the conspiracy theory classifier from relying on stylis- tic features specific media source for identify- ing conspiracy stories, will create media souree aware train dev test data splits for our experi- ments as well, addition to standard random data splitting that disregards data sources.
h
Σ 1 TER
(17)
The standard graph attention network is em- ployed to process event-doc relation, and update the document node embedding from connected events using the standard attention mechanism. The doc- ument node embedding at (-th layer is denoted as d()) During propagation, the attention weights Oi across the connected events are computed as:
4.2 Experimental Settings
We design two different settings: 1) identifying con- spiracy theories from unseen new media sources,
c LickyyRlll (a (Wal-M) @ Wh a exp(ei)
DD
(18)
2) from random media sources:
04 softmar;(cs) = Expx((e
(19)
Media Source Splitting: We split the dataset based on media sources, and articles from the same media source will not appear in the same set: training, development, or testing. This en- sures that the model is evaluated on articles whose sources were not seen during training. This setting will remove the inflation in per- formance that due the conspiracy theory classifier relying on the shorteuts of media sources features to make prediction. Table 2 and presents the statistics of media sources and articles within the train, dev, and test sets.
where C the events set, a and W are trainable pa- rameters. The final output feature for the document node at 1-th layer is calculated as:
1 Σ (a
(20)
This document node embedding is the final embed- ding to represent the whole article, which contains both the information of graph structure and article context. We further build two-layer classification head on top to predict conspiracy theories, and use cross-entropy loss for training.
Random Splitting: We adopt the standard

---

MUC
CEAF.
BLANC
Precision
Recall
F1 79.57
Precision
Recall
Precision
Recall
Precision 83.69
Recall
76.34
83.10
97.07
98.32
97.69
97.79
97.00
97.39
92.43
87.54
Table 4: Performance of event coreference resolution in the event relation graph
Precision
Recall
F1
all-conspiracy: naive baseline that catego- rizes all the documents into conspiracy class
Event Identifier
87.31
91.81
89,40
Table 5: Performance of event identification. Macro precision, recall, and F1 are reported.
chatgpt: where we designed an instruction prompt (A.1) that allows the large language model ChatGPT to automatically generate pre- dicted labels for each news article within the same test
Precision
Recall
Temporal 48.45
46.43
47.04
Causal
58.48
54.02
56.01
Subevent 53.37
42.90
46.21
where we use the same language model Longformer (Beltagy et al., 2020) and add an extra layer of Bi-LSTM on top. The embedding of the article start token is used article embedding. The same two-layer classi- fication head is built on top of the article em- bedding to predict conspiracy theories. This baseline model is equivalent to our developed model without event relation graph.
longformer:
Table Performance of temporal, causal, and subevent relation tasks in the event relation graph. Macro preci- sion, recall, and FI are reported.
method to split train / dev test set randomly. For fair comparison, we ensure the equal size of training and development set, and also eval- uate on the same testing set used the media source splitting. In this setting, articles from the same media source can appear in both training and evaluation sets.
longformer additional features: where we concatenate the soft labels eq (1)-(5) addi- tional features into the article embedding.
4.3
Event Relation Graph
4.5 Experimental Results
The event relation graph is trained on MAVEN- ERE dataset (Wang et al., 2022) which contains massive general-domale news articles. The current state-of-art model framework (Wang al., 2022) is adopted learn different components collabo- ratively. The performance of event identification is presented in Table 5. Table shows the results of event coreference relation identification. Fol- lowing the previous work (Cai and Strube, 2010), MUC (Vilain et al., 1995), B3 (Bagga and Bald- win, 1998), CEAFe (Luo, 2005), and BLANC (Recasens and Hovy, 2011) used evaluation metrics. Table shows the performance of other components in event relation graph, including tem- poral, causal, and subevent relation tasks. The standard macro-average precision, recall, and FI score are used for evaluation.
Table 7 reports the experimental results of conspir- acy theories news identification under different splitting settings. Precision, recall, and F1 score of the conspiracy class is shown.
In the media source splitting setting, we can see that incorporating the event relation graph sub- stantially boosts precision by 3.59% and recall by 5. 13,,, compared to the longformer baseline. This suggests that the event relation graph encapsulates events and their logical interrelations, thereby has the ability understand complex narratives of con- spiracy theories. These improvements also show that our proposed method can generalize well for unseen new media sources.
In the random splitting setting, incorporating the event relation graph can also bring significant im- provement to both precision and recall. The results indicate the effectiveness of the event relation graph method under either the hard and easy setting, with the Fl score increased by 4.22% 4.47%. Un- surprising, the system performance in this setting is overall higher than in the media source split- ting, probably due to the inflation caused by the
4.4 Baselines
Our paper presents the first attempt to identify con- spiracy theories in news articles. There are few established methods available for comparison. We experimented the following systems as baselines:

---

Data Split Settings
Media Source Splitting
Random Splitting
Precision
Recall
F1
Precision
Recall
F1
Bascline Model
all-conspiracy
23.68
100.00
38.30
23.68
100.00
38.30
chatgpt
64.77
28.84
39.91
64.77
28.84
39.91
longformer
79.53
69.32
74.08
86.59
75.14
80.46
longformer + additional features
79.37
70.33
74.58
86.83
75.90
80.99
Event Relation Graph
event-aware language model (soft label)
82.30
70.27
75.81
89.73
77.92
83.41
event relation graph encoder (hard label)
79.92
73.24
76.44
88.88
80.39
84.42
both (full model)
83.12
74.45
78.55
89.22
80.58
84.68
Table 7: Experimental results of conspiracy theories news identification based on event relation graph under two different splitting settings. Precision, Recall, and F1 of the positive class are shown. The model with the best performance is bold.
classifier taking the shortcut of using media source recognition for conspiracy theory detection.
Precision
Recall
F1
baseline full model
79.53
69.32
74.08
The event relation graph method outperforms the simple feature concatenation baseline. We ob- serve that compared with the longformer baseline, incorporating probabilistic vectors as additional features (longformer additional features) only slightly improve the performance. This demon- strate that developing more sophisticated methods to fully leverage the information embedded within the event relation graph is necessary.
83.12
74.45
78.55
event identify
82.75
73.12
77.64
coreference
82.64
72.87
77.45
temporal
80.15
73.06
76.44
causal
82.49
72.11
76.95
subevent
82.11
72.30
76.89
Table 8: Effect of removing each the event relation graph components: event identification, coreference, temporal, causal, and subevent relations.
The method based on event relation graph per- forms significantly better than the chatgpt. Com- paring our full model to the chatgpt baseline, there is still a large gap in performance, especially the recall. We observe that chatgpt chooses conspiracy label mostly when false narratives are explicit. On the contrary, the event relation graph concentrates logical reasoning, thus better deal with im- plicit cases and yields higher recall.
and ultimately improving precision. The utilization of soft labels and hard labels are complementary with each other. Incorporating both soft labels and hard labels (the full model) exhibit the best perfor- mance.
4.7 Effect of Different Event Relations
4.6 Ablation Study
The ablation study is also shown in Table 7. The event relation graph encoder based on hard labels contributes enhancing recall, while the event- aware language model leveraging soft labels con- tributes to improving precision. It is probably be- cause that the hard labels employ the four event relations in relatively aggressive manner, by con- structing the corresponding relation regardless of the confidence level. This enables the encoder to fully exploit the intrinsic information embedded in the event relation graph, thereby enhancing re- call. On the contrary, the soft labels incorporate predicted probabilities of the event relation graph, allowing the model to learn from confidence levels
We further study the effect of the different event relations in the event relation graph. Table 8 shows the experimental results of removing each type of event relations from full model. The results show that removing any type of event relations leads to a performance drop, reducing both pre- cision and recall. Therefore, each type of event relations plays an essential role in constructing a comprehensive content structure, and is crucial for identifying conspiracy theories news. Further, the experimental results demonstrate the importance of different event relations: removing temporal, causal, or subevent relations leads to larger perfor- mance decrease.

---

successfull oxample
vaceinstion
cutune
uutism
Taking vaceination can cause autism in childrren.. Yates Hazlehurst, a yoar-old boy, suffered indemark brain discase, pain, and inability to communicate,, all of which are typical of severe
reaction after taking vaceinations, including
sunglorence
sucinsel vaccination
discase
(sunspiracy baseline)
P (conspiracy event logic graph) - 0.76
after
pain
rcaction
inability
failing example
COVID-19 pandemic has something do with the newly emergene 5G technology... ... The electronic device inside 5G circuit board actually labeted "COV-19", which is proof the discssc outbreak connected the newly emergent cellular technelogy.
pundernic
casseed by teechooooyy
P (connspiracy haseline) - 0.24
conffonance
(conspiracy event logie gruph) 0.49
technology:
An example of our method succeed in identifying conspiracy theory news, and a failing example. Events are shown in bold text. The solid arrows in the event relation graphs represent the successfully extracted event relations, and the dashed arrow means the missing event relation.
Figure 3:
4.8 Analysis and Discussion
sults demonstrate the effectiveness of our approach. Future work needs to develop more sophisticated methods to improve event relation graph construc- tion and better extract implicit event relations.
Figure shows an example where our method suc- ceeds in solving false negative. By identifying the events and extracting the relations between events, the model is aware of the author's intention to at- tribute a causal relation between taking vaccination and autism. At the same time, the model is encoded with the information that the reaction event consist- ing of disease, pain, and inability happened after taking vaccination, which may not be adequate to reach a causal conclusion. The model not only learns the overall distribution of the four event re- lations, but is also aware of the specific relations described within story, thereby can better com- prehend complex narratives.
Limitations
Our paper proposes to build an event relation graph for each article to facilitate conspiracy theories un- derstanding. The analysis shows that the current event relation graph algorithm has the ability extract event relations for most easy cases, but fail in recognizing implicitly stated relations. order to further improve the performance of con- spiracy theorics identification, develop more sophisticated methods to enhance the performance of event relation graph.
necessary to
An example where our method fails to identify conspiracy theories also shown in Figure 3. The author intends to imply a causal relation between COVID-19 pandemic and emergent 5G technology. However, the text expresses this relation in a very implicit way by using the phrase has something to do with. The event relation graph algorithm fails recognize this implicitly stated causal relation and leads to a false negative error. In order to fur- ther improve the performance of conspiracy theory identification, it is necessary to improve event rela- tion graph construction and better extract implicit event relations.
Ethics Statement
This paper aims to identify conspiracy theories, which is a specific form of misinformation. De- tecting such misinformation can promote critical thinking, mitigate misinformation, preserve trust, and safeguard societal well-being. The conspiracy theories dataset is used only for academic research purpose. The release of conspiracy theories dataset and code should only be utilized for the purpose of combating misinformation, and should not be used for spreading misinformation.
5 Conclusion
Acknowledgements
This paper presents the first attempt to identify con- spiracy theories in news articles. Following our ob- servation that conspiracy theories be fabricated by mixing uncorrelated events together or present- ing an unusual distribution of relations between events, we construct an event relation graph for each article and incorporate the graph structure for conspiracy theories identification. Experimental re-
We would like to thank the anonymous reviewers for their valuable feedback and input. We grate- fully acknowledge support from National Science Foundation via the awards IIS-1942918 and IIS- 2127746. Portions of this research were conducted with the advanced computing resources provided by Texas A&M High- Performance Research Com- puting.

---

References
Karen M Douglas, Joseph E Uscinski, Robbie M Sut- ton, Aleksandra Cichocka, Turkay Nefes, Chee Siang Ang, and Farzin Deravi. 2019. Understanding con- spiracy theories. Political psychology, 40:3-35.
Kiril Avramov, Vasily Gatov, and Ilya Yablokov. 2020. Conspiracy theories and fake news. In Rouiledge Handbook of Conspiracy Theories. Routledge.
Lisa Fan, Marshall White, Eva Sharma, Ruisi Su, Pra- fulla Kumar Choubey, Ruihong Huang. and Lu Wang. 2019. In plain sight: Media bias through the lens of factual reporting. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages $343-6349 Hong Kong, China. Association for Computational Linguistics.
Amit Bagga and Breck Baldwin. 1998. Entity-based cross-document coreferencing using the vector space model. In COLING 1998 Volume I: The 17th Inter- national Conference on Computational Linguistics.
Ramy Baly, Giovanni Da San Martino, James Glass, and Preslav Nakov. 2020. We can detect your bias: Predicting the political ideology of news articles. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4982-4991, Online. Association for Computa- tional Linguistics.
Neophytos Georgiou, Paul Delfabbro, and Ryan Balzan. 2021. Conspiracy theory belief', scientific reasoning and the analytical thinking paradox. Applied Cogni- rive Psychology, 35(6):1523-1555
Iz Beltagy, Matthew E. Peters, and Arman Cohan, 2020. Longformer: The long-document transformer.
Nacemul Hassan, Fatma Arslan, Chengkai Li, and Mark Tremayne. 2017. Toward automated fact-checking: Detecting check-worthy factual claims by claim- buster. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1803-1812.
Steven Bethard, Oleksandr Kolomiyets, and Marie- Francine Moens. 2012. Annotating story timelines as temporal dependency structures. In Proceed- ings of the Eighth International Conference on Lan- guage Resources and Evaluation (LREC'12), pages 2721-2726 Istanbul, Turkey. European Language Resources Association (ELRA).
Pavan Holur, Tianyi Wang. Shadi Shahsavari, Timo- thy Tangherlini, and Vwani Roychowdhury. 2022. Which side you on? insider-outsider classification in couspiracy conspiracy-theoreiic theoretic social media. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages -975-4987, Dublin, Ireland. Association for Computational Linguistics.
Paul Bleakley. 2021. Panic, pizza and mainstreaming the alt-right: A social media analysis of pizzagate and the rise of the qanon conspiracy. Current Sociology. page 001 (113122121499999
Jie Cai and Michael Strube. 2010. Evaluation metrics for end-to-end coreference resolution systems. In Proceedings of the SIGDIAL 2010 Conference, pages 28-36. Tokyo, Japan. Association for Computational Linguistics.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirec- tional Istm-erf models for sequence tagging.
J.P. Hughes, A. Efstratiou, S.R. Komer, L.A. Baxter. M. Vasiljevic, and A.C. Leite. 2022. The impact of risk perceptions and belief in conspiracy theories on covid- 19 pandemic-related behaviours. PLoS One, 17:c0263716.
Matteo Cinelli, Gabriele Etta, Michele Avalle, Alessan- dro Quattrociocchi,, Niccolò Di Marco, Carlo Valen- sise, Alessandro Galeazzi, and Walter Quattrociocchi. 2022. Conspiracy theories and social media plat- forms. Current Opinion in Psychology, page 101407.
Zhiwei Jin, Juan Cao. Han Guo, Yongdong Zhang, and Jicbo Luo. 2017. Multimodal fusion with recurrent neural networks for rumor detection on microblogs. In Proceedings of the 25th ACM international con- ference on Multimedia, pages 795-816.
Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeñoo, Rostislav Petrov, and Preslav Nakov. 2019. Fine-grained analysis of propaganda in news article. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing BEMNLP-IJCNLP pages 5636-5646, Hong Kong, China. Association for Com- putational Linguistics.
Neal Kumar Katyal. 2002. Conspiracy theory. Yale Lj. 112:1307.
C. Klein, P. Clutton, and A. G. Dunn. 2019. Pathways to conspiracy: The social and linguistic precursors of involvement in reddit's conspiracy theory forum. PLOS ONE, 14(1 ec0225099.
David De Coninek, Thomas Frissen, Koen Matthijs, Leen d' H'Hennenss Grégoire Lits Olivier Champagne- Poirier, Marie-Eve Carignan. Mare D David, Nathalie Pignard-Cheynel. Sébastien Salerno, et 2021. Be- liefs in conspiracy theories and misinformation about covid- 19: Comparative perspectives on the role of anxiety, depression and exposure to and trust in infor- mation sources. Frontiers in psychology, 12:646394.
Yuanyuan Lei and Ruihong Huang. 2022. Few-shot (dis)agreement identification in online discussions with regularized and augmented meta-learnin. In Findings of the Association for Computational Lin- guisties: EMNLP 2022, pages 5581-5593, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.

---

Yuanyuan Lei, Ruihong Huang, Lu Wang. and Nick Beauchamp. 2022. Sentence-level media bias analy- sis informed by discourse structures. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10040-10050, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Samantha Phillips, Lynnette Hui Xian Ng, and Kath- leen M Carley. 2022. Hoaxes and hidden agendas: A twitter conspiracy theory dataset: Data paper. In Companion Proceedings of the Web Conference 2022, pages 876-880.
Marta Recasens and Eduard Hovy. 2011. Blanc: Im- plementing the rand index for corfference evaluation. Natural language engineering. 17(4):485-510.
Marie-Jeanne Leonard and Frederick I.. Philippe. 2021. Conspiracy theories: public health concern and how to address it. Frontiers in Psychology, 12:682931.
Victoria I. Rubin, Yimin Chen, and Nadia K Conroy. 2015. Deception detection for news: three types of fakes. Proceedings of the Association for Informa- lion Science and Technology, 52(1):1-4.
Xiaoqiang Luo. 2005. On coreference resolution perfor- mance metrics. In Proceedings of Human Language Technology Conference and Conference on Empiri- cal Methods in Natural Language Processing, pages 25-32, Vancouver, British Columbia, Canada. Asso- ciation for Computational Linguistics.
Mattia Samory and Tanushree Mitra. 2018. 'the gov- ernment spies using our webcams' the language of conspiracy theories in online discussions. Proceed- ings of the ACM on Human- Coman-Cmmputer Interaction. 2(CSCW):1-24
Kai Shu, Amy Sliva, Suhang Wang. Jiliang Tang, and Huan Liu. 2017. Fake news detection on social me- dia: A data mining perspective.
Jing Ma. Wei Gao, Prasenjit Mitra, Sejeong Kwon, Bernard J Jansen, Kam-Fai Wong, and Meeyoung Cha. 2016. Detecting rumors from microblogs with recurrent neural networks. AAAI Press.
N. Smith and T. Graham. 2019. Mapping the anti- vaccination movement facebook. Information, Communication & Society, 2210):111-
Silvia Mari, Homero Gil de Zuniga, Ahmet Suerdem, Katja Hanke, Gary Brown, Roosevelt Vilar. Diana Boer, and Michal Bilewiez. 2022. Conspiracy the- ories and institutional trust: examining the role of uncertainty avoidance and active social media use. Political Psychology, 43(2):277-226.
Fiona Anting Tan, Ali Hürriyetollu, Tommaso Caselli, Nelleke Oostdijk, Tadashi Nomoto, Hansi Het- tiarachchi, Iqra Ameer, Onur Uca, Farhana Ferdousi Liza, and Tiancheng Hu. 2022. The causal news cor- pus: Annotating causal relations in event sentences from news. In Proceedings of the Thirteenth Lan- guage Resources and Evaluation Conference, pages 231-, Marseille, France. European Language Resources Association.
Priyanka Meel and Dinesh Kumar Vishwakarma. 2020. Fake news, rumor, information pollution in social me- dia and web: A contemporary survey of state-of-the- arts, challenges and opportunities. Expert Systems with Applications, 153:11290
Timothy R Tangherlini, Shadi Shahsavari, Behnam Shahbazi, Ehsan Ebrahimzadeh, and Vwani Roy- chowdhury. 2020. An automated pipeline for the dis- covery of conspiracy and conspiracy theory narrative frameworks: Bridgegate, pizzagate and storytelling the web. PloS one, 15())cc233389.
Alessandro Miani, Thomas Hills, and Adrian Bangerter. 2021. Loco: The 88-million-ue word language of con- spiracy corpus. Behavior research methods, pages 1 1-24.
Seong Jae Min. 2021. Who believes in conspiracy theo- ries? network diversity, political discussion, and con- servative conspiracy theories on social media. Amer- ican Politics Research, 495--444
James Thome and Andreas Vlachos. 2018. Automated fact checking: Task formulations, methods and fu- ture directions. In Proceedings of the 27th Inter- national Conference Computational Linguistics, pages 3346-3359, Santa Fe, New Mexico, USA. As- sociation for Computational Linguistics.
J. D. Moffitt, C. King, and K. M. Carley, 2021. Hunting conspiracy theories during the covid-19 19 pandemic. Social Media + Society, 7(3).
Eiprian-cctavinn Truică and Alena-Simona Apostol. 2022. Misrobærta: Transformers versus misinfor- mation. Mathematics, 10(4):569.
Eric Oliver and Thomas J Wood. 2014. Conspiracy theories and the paranoid style (s) mass opinion. American journal of political science, 58(4):952- 966.
Sander Van der Linden. 2015. The conspiracy-effectt Exposure to conspiracy theories (about global warm- ing) decreases pro-social behavior and science ac- ceptance. Personality and Individual Differences. 87:171-173.
Verónica Pérez-Rosiee Bennett Kleinberg, Alexandra Lefevre, and Rada Mihalcea. 2018 Automatic de- tection of fake news. In Proceedings of the 27th International Conference on Computational Linguis- tics, pages 3391-3401, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Jan-Will Willemll van Prooijen and Karen M. Douglas. 2018. Belief in conspiracy theories: Basic principles of an emerging research domain. Europenn Journal of Social Psychology, 48:897-908.

---

Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph attention networks.
A
Appendix
A.1
ChatGPT Prompt
The designed instruction prompt for ChatGPT base- line is: "Conspiracy theories are narratives that explains event situation in irrational malicious manner. Please reply Yes if the follow- ing text contains conspiracy theory, else reply No. Text: xxx. Answer:"
Mare Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model- theoretic coreference scoring scheme. In Sixth Mes- sage Understanding Conference (MUC-6); Proceed- ings of Conference Held in Columbia, Maryland, November 6-8, 1995.
Xiaozhi Wang, Yulin Chen, Ning Ding, Hao Peng, Zimu Wang. Yankai Lin, Xu Han, Lei Hou, Juanzi Li, Zhiyuan Liu, Peng Li, and Jie Zhou. 2022. MAVEN- TRE: A unified large-scale dataset for event coref- erence, temporal, causal, and subevent relation ex- traction. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 926-941, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Xiaozhi Wang. Ziqi Wang, Xu Han, Wangyi Jiang. Rong Han, Zhiyuan Liu, Juanzi Li, Peng Li, Yankai Lin, and Jie Zhou. 2020. MAVEN: A Massive General Domain Event Detection Dataset. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1652- 1671, Online. Association for Computational Linguis- tics.
M. J. Wood. 2018. Propagating and debunking conspir- acy theories on twitter during the 2015-2016 zika virus outbreak. Cyberpsychology, Behavior, and So- cial Networking, 21 (8)5:4855490.
Michael J. Wood and Karen M. Douglas. 2013. "what about building 7?" social psychological study of on- line discussion of 9/11 conspiracy theories. Frontiers in Psychology, 4.
Wenlin Yao, Zeyu Dai, Maitreyi Ramaswamy, Bonan Min, and Ruihong Huang. 2020. Weakly Supervised Subevent Knowledge Acquisition. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5345- 5356, Online. Association for Computational Lin- guistics.
Yutao Zeng. Xiaolong Jin, Saiping Guan, Jiafeng Guo, and Xueqi Cheng. 2020. Event coreference resolu- tion with their paraphrases and argument-awand em- beddings. In Proceedings of the 28th International Conference Computational Linguistics, pages 3084-3094, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Xiyang Zhang, Muhao Chen, and Jonathan May. 2021. Salience-awer aware event chain modeling for narrative understanding. In Proceedings of the 202/ Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 1418-1428, Online and Punta Cana, Dominican Republic, Association for Computational Linguistics.


Reviews:
[{"review_id": "cYpbmJPA7G", "reviewer": "Reviewer_nzA4", "summary": "", "questions": "What is the training time and the parameters you used or evaluated to reach the results reported in the paper?\n\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments! Here are our answers (A) to your questions (Q):\n\n**Q1:** The reproducibility of the paper could be strengthened by providing dependencies/requirements of the code submitted with this report.\n\n**A1:** We will provide detailed instructions for the released version of code.\n\n**Q2:** What is the training time and the parameters you used or evaluated to reach the results reported in the paper?\n\n**A2:** The training time of our model is around 8 hours on one GPU of Nvidia RTX 3090. The AdamW (Loshchilov and Hutter, 2019) is used as the optimizer. The learning rate is initialized as 1e-5 and then adjusted by a linear scheduler. The weight decay is set to be 1e-2. The maximum length of input is set to 4096. The number of training epochs is 5. We will add these information into the paper.\n\n**Q3:** Missing References and Typos\n\n**A3:** We will add the references you suggested into the paper, and correct the typos.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1- The task is important. \n\n2- The task and the method are described well. \n\n3- The evaluation is realistic and shows that the proposed system achieves good performance.\n\n4- The paper is written well.", "reasons_to_reject": "The reproducibility of the paper could be strengthened by providing dependencies/requirements of the code submitted with this report.", "questions_to_authors": "What is the training time and the parameters you used or evaluated to reach the results reported in the paper?\n\n", "paper_topic_and_main_contributions": "The authors propose a method for identifying conspiracy theories that are based on an event graph constructed using events as nodes and event relations such as coreference, causality, temporal, and subevent as relations. This task is important and has not been studied much on long texts.  The method proposed in this report is compared to strong baselines. Moreover, the cross-source evaluation, in addition to random splitting, provides a realistic evaluation. The error analysis and ablation study enable a reader to understand both the task and merits of the proposed methodology for conspiracy detection in long text such as text.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "In addition to random splitting, using source-based splitting is great. I can suggest referring to studies like the ones below in this line: \n1- Hürriyetoğlu, A., Yörük, E., Mutlu, O., Duruşan, F., Yoltar, Ç., Yüret, D., & Gürel, B. (2021). Cross-context news corpus for protest event-related knowledge base construction. Data Intelligence, 3(2), 308-335.\n\n2- Hürriyetoğlu, A., Yörük, E., Yüret, D., Yoltar, Ç., Gürel, B., Duruşan, F., & Mutlu, O. (2019). A task set proposal for automatic protest information collection across multiple countries. In Advances in Information Retrieval: 41st European Conference on IR Research, ECIR 2019, Cologne, Germany, April 14–18, 2019, Proceedings, Part II 41 (pp. 316-323). Springer International Publishing.", "Typos_Grammar_Style_and_Presentation_Improvements": "1- \"through the url\" -> through the URL\n\n2- \"link in social media platforms\" -> on social media platforms\n\n3- studied in isolation(Zeng et al. -> studied in isolation (Zeng et al.\n\n4- The numbers could be formatted better, e.g. 14321 -> 14,321\n"}}, {"review_id": "bIcogyG0Pk", "reviewer": "Reviewer_9T2j", "summary": "", "questions": "This paper studied an interesting problem on detecting conspiracy and provided some useful insights into the problem. I have a few questions.\n\n1. The authors highlighted that most previous studies on conspiracy detection have focused on social media short texts. It would be useful to explain how previous methods on short texts cannot work well on long news articles. This also relates to the choice of baselines in the experiments. As the dataset was released recently, there might not be available baselines that have been directly applied to the dataset. I think it would be useful to include some state-of-the-art baselines that were used for short text in this case --- to show the superiority of the proposed method but also show the differences between short and long text domains. The current baselines look weak: one is a \"reduced\" version of the proposed method, one is a general unsupervised language model, and another one is even not a genuine classifier. \n\n2. I found a bit hard to follow the section 3.2 on event-ware language model and I am not sure why this step is needed. I guess that this step is to incorporate both contextual information in a raw document and event relations extracted from section 3.1. It would be useful to make it clearer. \n\n3. Although the experimental results show that the hard labels are useful, I feel that hard labels are redundant, as they are a transferred version of soft labels using the argmax function. In fact, the soft labels might provide rich information due to their fuzzy/probability nature, particularly when there is ambiguity (or overlapping between multiple relations) between a pair of events. Just a suggestion. I would explore whether it is possible to use a vector <P_event, P_corefer, P_temp, ..... ,> as weights or representation of an edge between events, and drop hard labels to make the whole process neater. \n\n4. In Figure 3, events that have a relation of coreference often share the same string. Should we collapse these same strings as a single node to reduce the size of graph and training time?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable suggestions and questions! Here are our answers (A) to your questions (Q):\n\n\n**Q1:** Baselines were slightly weak. The authors highlighted that most previous studies on conspiracy detection have focused on social media short texts. I think it would be useful to include some state-of-the-art baselines that were used for short text in this case  --- to show the superiority of the proposed method but also show the differences between short and long text domains.\n\n**A1:** We do not present previous work on social media short text as baselines because most of previous work discuss psychology or communication perspectives of conspiracy theories from social media, instead of developing computational models to detect them.\n- **Most previous work studied conspiracy theories from psychology or communication perspectives, and few work focused on computational detection model building.** The work such as *De Coninck et al. 2021*, *Min Seong Jae 2021*, *Mari Silvia et al. 2022* studied the problem of who believes in conspiracy theories, from psychology perspective. The work such as *Wood 2018*, *Smith and Graham 2019*, *Klein et al 2019* researched the dissemination pattern of conspiracy theories on social media platforms, from a communication perspective. To the best of our knowledge, *Phillips et al. 2022* is the only work that engaged with computational model to detect conspiracy theories in social media short text.\n- To alleviate your concern, we build an additional baseline based on the method released in *Phillips et al. 2022*, and the results demonstrate that **the model developed for social media short text is not capable to detect conspiracy theories in long news articles.** We show the results under two experimental settings in the first row of the table below. There exists a noticeable performance gap between the model (Phillips et al. 2022) developed for social media short text and our longformer baseline developed for news. This indicates the different characteristics between social media short text and news articles. Understanding lengthy news articles requires profound comprehension towards news structure and complicated narratives.\n\n\n| Data Split Settings | Media Source Splitting | Random Splitting |\n| ------------------- | ---------------------- | ---------------- |\n|     | Precision / Recall / F1    | Precision / Recall / F1 |\n| *Phillips et al. 2022 (baseline)* | 57.12 / 48.88 / 52.67 | 66.01 / 53.62 / 59.17 |\n| longformer (baseline) | 79.53 / 69.32 / 74.08 | 86.59 / 75.14 / 80.46 |\n| event relation graph (our method) | 83.12 / 74.45 / 78.55 | 89.22 / 80.58 / 84.68 |\n\n\n>[1] De Coninck, David, et al. \"Beliefs in conspiracy theories and misinformation about COVID-19: Comparative perspectives on the role of anxiety, depression and exposure to and trust in information sources.\" Frontiers in psychology 12 (2021): 646394.\\\n>[2] Min, Seong Jae. \"Who believes in conspiracy theories? Network diversity, political discussion, and conservative conspiracy theories on social media.\" American Politics Research 49.5 (2021): 415-427.\\\n>[3] Mari, Silvia, et al. \"Conspiracy theories and institutional trust: examining the role of uncertainty avoidance and active social media use.\" Political Psychology 43.2 (2022): 277-296.\\\n>[4] Wood, Michael J. \"Propagating and debunking conspiracy theories on Twitter during the 2015–2016 Zika virus outbreak.\" Cyberpsychology, behavior, and social networking 21.8 (2018): 485-490.\\\n>[5] Smith, Naomi, and Tim Graham. \"Mapping the anti-vaccination movement on Facebook.\" Information, Communication & Society 22.9 (2019): 1310-1327.\\\n>[6] Klein, Colin, Peter Clutton, and Adam G. Dunn. \"Pathways to conspiracy: The social and linguistic precursors of involvement in Reddit’s conspiracy theory forum.\" PloS one 14.11 (2019): e0225098.\\\n>[7] Holur, Pavan, et al. \"Which side are you on? Insider-Outsider classification in conspiracy-theoretic social media.\" In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4975–4987, Dublin, Ireland. Association for Computational Linguistics.\\\n>[8] Phillips, Samantha C., Lynnette Hui Xian Ng, and Kathleen M. Carley. \"Hoaxes and hidden agendas: A twitter conspiracy theory dataset\" Companion Proceedings of the Web Conference 2022. 2022.\n\n\n\n\n**Q2:** Just a suggestion. I would explore whether it is possible to use a vector <P_event, P_corefer, P_temp, ..... ,> as weights or representation of an edge between events.\n\n\n**A2:** Thanks for your suggestion. We follow your suggestion to add another baseline of concatenating the soft label vector <P_event, P_corefer, P_temp, P_causal, P_subevent> as additional features into the article embedding. The results under two experimental settings are reported in the second row of the table below. We observe that, compared with the longformer baseline, incorporating soft labels as additional features only slightly improve the performance. One possible explanation is that simple feature concatenation cannot fully exploit the information embedded within the event relation graph. **The results demonstrate that our event relation graph approach still performs better than the baseline with probability features.**\n\n\n| Data Split Settings | Media Source Splitting | Random Splitting |\n| ------------------- | ---------------------- | ---------------- |\n|     | Precision / Recall / F1    | Precision / Recall / F1 |\n| longformer (baseline) | 79.53 / 69.32 / 74.08 | 86.59 / 75.14 / 80.46 |\n| *longformer + additional features* | 79.37 / 70.33 / 74.58 | 86.83 / 75.90 / 80.99 |\n| event relation graph (our method) | 83.12 / 74.45 / 78.55 | 89.22 / 80.58 / 84.68 |\n\n\n\n\n**Q3:** I found a bit hard to follow the section 3.2 on event-ware language model and I am not sure why this step is needed. I guess that this step is to incorporate both contextual information in a raw document and event relations extracted from section 3.1. It would be useful to make it clearer.\n\n\n**A3:** **The purpose of event-aware language model is to distill the events and event relations knowledge from soft labels into the fundamental language model.** We build neural networks on top of the fundamental language model, guiding it to learn from soft labels. Therefore, the events and event relations knowledge within the soft labels can be distilled into the fundamental language model, resulting in an event-aware language model. Without this event-aware language model, the fundamental language model does not know that nodes in the graph are events and links are event relations.\n\n\n\n\n\n**Q4:** Although the experimental results show that the hard labels are useful, I feel that hard labels are redundant, as they are a transferred version of soft labels using the argmax function. In fact, the soft labels might provide rich information due to their fuzzy/probability nature, particularly when there is ambiguity (or overlapping between multiple relations) between a pair of events.\n\n\n**A4:** The hard labels are essential in our proposed method. We would like to make several points about the benefits of incorporating hard labels that can not be achieved by incorporating soft labels:\n- **The hard labels enable us to represent the event relation graph as a graph structure.** The event-aware language model based on the soft labels has events and event relations knowledge, but does not have the graph structural information. The hard labels enable us to explicitly link the event nodes through four relations, and eventually form a graph structure.\n- **The hard labels enable us to update event embeddings with their neighbor events information.** The graph structure constructed based on the hard labels enables us to update the event embeddings with their neighbor events information through connected relations. (section 3.3 relation-aware graph attention network)\n- **The hard labels enable us to derive a final graph embedding for conspiracy theories detection.** The graph structure facilitates us to derive a final graph embedding to represent the document, which is further used for conspiracy theories detection, while soft labels can not. (section 3.3 standard graph attention network)\n\n\n\n\n**Q5:** In Figure 3, events that have a relation of coreference often share the same string. Should we collapse these same strings as a single node to reduce the size of graph and training time?\n\n\n**A5:** **The two events with the same trigger word can have different context**, such as event arguments and other contextual information, **and thus should have different embeddings**. Therefore, we kept a node for each event mention in the graph.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Interesting research question; Rich details; promising results.", "reasons_to_reject": "Baselines were slightly weak; some steps are unclear;", "questions_to_authors": "This paper studied an interesting problem on detecting conspiracy and provided some useful insights into the problem. I have a few questions.\n\n1. The authors highlighted that most previous studies on conspiracy detection have focused on social media short texts. It would be useful to explain how previous methods on short texts cannot work well on long news articles. This also relates to the choice of baselines in the experiments. As the dataset was released recently, there might not be available baselines that have been directly applied to the dataset. I think it would be useful to include some state-of-the-art baselines that were used for short text in this case --- to show the superiority of the proposed method but also show the differences between short and long text domains. The current baselines look weak: one is a \"reduced\" version of the proposed method, one is a general unsupervised language model, and another one is even not a genuine classifier. \n\n2. I found a bit hard to follow the section 3.2 on event-ware language model and I am not sure why this step is needed. I guess that this step is to incorporate both contextual information in a raw document and event relations extracted from section 3.1. It would be useful to make it clearer. \n\n3. Although the experimental results show that the hard labels are useful, I feel that hard labels are redundant, as they are a transferred version of soft labels using the argmax function. In fact, the soft labels might provide rich information due to their fuzzy/probability nature, particularly when there is ambiguity (or overlapping between multiple relations) between a pair of events. Just a suggestion. I would explore whether it is possible to use a vector <P_event, P_corefer, P_temp, ..... ,> as weights or representation of an edge between events, and drop hard labels to make the whole process neater. \n\n4. In Figure 3, events that have a relation of coreference often share the same string. Should we collapse these same strings as a single node to reduce the size of graph and training time?", "paper_topic_and_main_contributions": "This paper studied the problem of detecting conspiracy theories, a type of misinformation that explains an event or situation in an irrational or malicious manner, in a news article. The authors proposed to construct a event relation graph for each article and incorporate the graph structure to identify conspiracy theories. Experimental results show good results of the proposed method.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "\"articles from the same media source will not appear in the same set\" -- > \"articles from the same media source will appear in the same set\"  in \"Media Source Splitting\" P6.\n\n\"89.73\" is higher than \"89.22\" and should be bold in Table 4."}}, {"review_id": "dJwa344N9B", "reviewer": "Reviewer_dygx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your reviews and comments! Here are our answers (A) to your concerns:\n\n\n**Q1:** What makes the ventilator story into a conspiracy theory is information that is far beyond the scope of what is presented in figure 1, namely: the huge-cover up that such a thing would entail, the known deadliness of covid-19, the general opinion of experts, which in turn is based on many years of scientific publications on ventilators, respiratory diseases, etc. Reiterating: it is information that is not in fig. 1 that leads us to conclude that the ventilator story is unlikely to be true. Maybe the LLM encodes some general-purpose knowledge, but we don't really know what role it plays here.\n\n\n**A1:** \n- We completely agree with you that conspiracy theories detection is a complicated task and may rely on the general-purpose knowledge encoded within the language model. However, we want to emphasize that **the baseline model we used is also able to utilize the knowledge encoded within the same language model. The only difference between our model and baseline is the extra knowledge from event relation graph.** The non-trivial F1 improvement under both media source splitting (4.47%) and random splitting (4.22%) has demonstrated the effectiveness of incorporating the event relation graph.\n- In particular, we would like to further explain the role of our event relation graph in conspiracy theory detection. We admit that detecting the causal relation between a death event and a treatment event may not be sufficient to recognize the article is expressing a conspiracy theory, instead, **the role of event relation detection is guiding the conspiracy theory detector and engaging its attention on significant event-event relations**, and so that the conspiracy theory detector can specifically check and reason (by making use of all the knowledge its language model can provide) if it is reasonable to see “treatment on ventilators” cause deaths.\n\n\n\n**Q2:** Then, I believe that it is telling that the F1 improvement over vanilla longformer is smaller than the difference between media and random splitting. I see no reason to assume that this model is detecting anything more than stylistic differences -- even with media split, conspiracy theories and benign news are written by different types of writers, with different types of audiences in mind, in different types of publications.\n\n\n**A2:** We do not deny that stylistic features play significant roles in detecting conspiracy theories, and this is why we perform experiments in two settings. But we do not see how the relatively large performance differences between the two settings will lead to the conclusion that our model is only detecting stylistic differences.\n- It is worthy to note that **the performance gains in the random splitting setting are solely from the features of the event relation graph.** Within the random splitting setting where both our model and baseline can leverage the media source stylistic features, incorporating the event relation graph features can improve F1 by 4.22% (80.46% -> 84.68%). This indicates that event relation graph features provide extra complementary information to the stylistic features.\n- **The event relation graph features can also improve performance when the stylistic features are not available or limited.** In order to remove or alleviate the impact of media source style features, we also design the media source splitting setting, where both our model and baseline can not use the media source stylistic features. The results show that incorporating event relation graph can also increase F1 by 4.47%, which proves the usefulness of the event relation graph when the stylistic features are limited.\n\n\nIn summary, the performance gains of our proposed model over baseline under both settings demonstrate the effectiveness of the event relation graph, whether the stylistic features are available or not.\n\n\n\n\n**Q3:** I am also not convinced that this model would perform so well with completely new topics . The fact that vanilla longformer already works so well raises the question that simply a certain combination of topics is enough to detect conspiracy theories with considerable accuracy. But what about completely new topics?\n\n\n**A3:**\n- **Our experiments demonstrate that the conspiracy theories detector relies more on the media source features than the topic features.** We did an experiment in the topic-based splitting setting, where the model detects conspiracy theories in unseen new topics. The experimental results are shown in the table below. Compared to the random splitting, mitigating the influence of media source features (media source splitting) leads to bigger performance drop than mitigating the influence of topic features (topic splitting). This phenomenon demonstrates that the conspiracy theories detector relies more on the media source features than the topic features. Therefore, we primarily focus on media source splitting in the paper due to the space limit.\n- **Our experiments indicate the effectiveness of event relation graph, whether media source features or topic features are available or not.** Comparing our proposed model (Row 4) with the baseline (Row 1), the performance gains indicate that incorporating event relation graph can increase precision and recall under media source splitting, topic splitting, and random splitting settings, regardless of the availability of media source features or topic features. We also did the ablation study of the two designed components under the topic-based splitting setting, as shown in the table (Row 2-3). We will add the results into the paper.\n\n\n\n| Data Split Settings | Media Source Splitting | Topic Splitting | Random Splitting |\n| ------------------- | ---------------------- | --------------- | ---------------- |\n| | Precision / Recall / F1 | Precision / Recall / F1 | Precision / Recall / F1 |\n| longformer (baseline) | 79.53 / 69.32 / 74.08 | 85.75 / 75.01 / 80.02 | 86.59 / 75.14 / 80.46 |\n| + event aware language model (soft label) | 82.30 / 70.27 / 75.81 | 88.58 / 78.36 / 83.16 | 89.73 / 77.92 / 83.41 |\n| + event relation graph encoder (hard label) | 79.92 / 73.24 / 76.44 | 86.17 / 81.40 / 83.72 | 88.88 / 80.39 / 84.42 |\n| + both (event relation graph full model) | 83.12 / 74.45 / 78.55 | 88.01 / 81.72 / 84.75 | 89.22 / 80.58 / 84.68 |\n\n\n\n**Q4:** I regret to say that I do not find any sufficiently significant contribution here.\n\n**A4:** Our primary **contributions** are summarized as follows for reference:\n\n* We are **the first paper to detect conspiracy theories news**. Although conspiracy theories are wide-spread and harmful, there are no developed computational models for detecting such misinformation yet, especially in long news documents.\n* We **find out events and event relations are crucial for conspiracy theories detection.** Our empirical observations and statistical analysis collectively affirm that conspiracy theories news can present an unusual connection between events -- departing from the typical *temporal* narrative structure but frequently relying on *causal* relations and *subevent* relations to convince the readers.\n* We **propose and build the event relation graph** to integrate four common event relations together. Furthermore, we also study the interactions and combination effect of the four relations, which were not well studied previously. We also design a new framework to incorporate the event relation graph into conspiracy theories detection.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This article focuses on an important contemporary topic of great public interest: the detection of conspiracy theories in news media. It uses a well justified and well designed modern NLP pipeline and it shows SOTA results in conspiracy theory detection in a human-annotated dataset of conspiracy theories and benign news.\n\nThe hypothesis that event relation knowledge graphs and reasoning about them is a crucial step in automatically identifying conspiracy theories is highly credible and worth pursuing.", "reasons_to_reject": "I agree with the authors on the importance of reasoning about event graphs, but unfortunately I find little evidence that this is happening in this method. I find that the author(s) take a too naive (and not rigorous view) on the idea of rationality. To illustrate: in the example with the ventilator deaths, it is claimed that attributing deaths to the ventilators is an attempt to convince the reader through an \"irrational chain of events\" (sic). The problem is that, going by the knowledge graph contained in figure 1 alone, there is no reason to consider attribution of deaths to ventilators irrational, just because they are meant to be a treatment. It unfortunately does happen that medical treatments kill patients, and it does happen that medical practices are later found to be counter-productive etc. What makes the ventilator story into a conspiracy theory is information that is far beyond the scope of what is presented in figure 1, namely: the huge-cover up that such a thing would entail, the known deadliness of covid-19, the general opinion of experts, which in turn is based on many years of scientific publications on ventilators, respiratory diseases, etc etc. If you presented a rational person the graph of figure 1 in abstract, claiming that \"there was a disease, there was a treatment, the treatment ended up being what killed the patients\", this could be true. Reiterating: it is information that is not in fig. 1 that leads us to conclude that the ventilator story is unlikely to be true. Maybe the LLM encodes some general-purpose knowledge, but we don't really know what role it plays here, if any.\n\nThen, I believe that it is telling that the F1 improvement over vanilla longformer is smaller than the difference between media and random splitting. I see no reason to assume that this model is detecting anything more than stylistic differences -- even with media split, conspiracy theories and benign news are written by different types of writers, with different types of audiences in mind, in different types of publications. I am also not convinced that this model would perform so well with completely new topics. The fact that vanilla longformer already works so well raises the question that simply a certain combination of topics is enough to detect conspiracy theories with considerable accuracy. But what about completely new topics?\n\nI regret to say that I do not find any sufficiently significant contribution here.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this work a model is proposed to identify conspiracy theories in news articles. The model design is based on the hypothesis that conspiracy theory identification in articles requires (or is greatly aided by) taking into account the events and their relationships as described in a text. A joint learning framework is thus adopted to extract graphs of events and their relations, which are classified as either: coreference, temporal, causal or subevent. Subsequently, the Longformer LLM (with an attention mechanism suitable for long text) is augmented with events and event relations knowledge with extra NNs to predict the classifications from the previous step, resulting in what the author(s) describe as an event-aware language model. Finally, the document token the event-aware LM and the tokens from the events are combined in a labelled graph (labelled by event relations / event-document link) that is encoded by a graph neural network, that feeds the classifier that labels the document as conspiracy theory or benign.\n\nThe event relations graph extractor is trained with the general-domain MAVEN-ERE dataset and the subsequent classifier pipeline is trained and tested with the LOCO conspiracy theory dataset. The percentages of singleton, temporal, causal and subevent cases found on the conspiracy set is found to be significantly different from the benign set, with causal and subevent connections more common in conspiracy and temporal more common in benign.\n\nThe approach presented in this work is compared against a classifier directly on top of Longformer (taking as input the article start token) and with ChatGPT prompts. The comparisons are done in two sets: by random splitting of LOCO and by media source splitting. The latter is to prevent the model from identifying artifacts of a certain publication source. In fact, both the model presented here and vanilla Longformer perform better on random split than media source split. Overall, the event relation graph approach performs much better then ChatGPT and somewhat better than vanilla Longformer on both comparisons (aprox 4.0 improvement of F-score in both cases).\n\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a computational model to detect what the authors describe as conspiracy theories in news articles. The authors hypothesize that “conspiracy theories” will contain what they describe as “uncorrelated events” or an “unusual distribution of relations between events.”  Based on this hypothesis, they propose a model which attempts to use both the events in a news article (e.g. an infected event or hospitalization event) and the relationships between such events (e.g. infected before hospitalization) to detect conspiracy theories in news articles. The authors evaluate their model on an existing dataset LOCO (Miani et al., 2021), which appears to use credible news sources and conspiracy news sources to label individual news documents. The authors also emphasize that their work is different from prior studies, which focus on detecting conspiracies in shorter social media posts. 

Overall, the reviewers offered a mixed assessment of this work. 

Reviewer nzA4 offered a shorter review emphasizing the importance of the task, the quality of the exposition and the soundness of the experiments. This reviewer also praised the authors for testing their model with source-based splitting (i.e., the model had to predict conspiracies in unseen news sources), which seems important for checking that the model can actually learn to detect substantive markers of conspiracy theories instead of just stylistic features of conspiracy news sources. This reviewer gave the paper a soundness score of 4 and an excitement score of 4.

Reviewer 9T2j also believed the experiments were sound (awarding a score of 3). But they were less excited about thew work than reviewer nzA4, awarding an excitement score of 3. In their review, 9T2j asked for additional evidence that models designed for shorter texts would perform less well on longer news articles. In response, the authors added an additional baseline in the rebuttal period. They showed that this method had F1 scores that were nearly 20 points lower than models designed for longer text. Reviewer 9T2j also asked about if it was necessary to include hard labels in the model, and the authors discussed their role in their rebuttal. This discussion appears less relevant because the authors compared these approaches empirically in Table 4. Reviewer 9T2j did not respond to the rebuttal. 

Reviewer dygx offered lower scores, and asked questions about the nature of conspiracy theories in their review. They hypothesized about the kinds of reasoning that might lead people to believe conspiracy theories, and the extent to which LLMs might already have representations of such reasoning. These comments from dygx were just a hypothesis without grounding in experiments or prior work, and should not be given too much weight. But the idea of a “conspiracy theory” could have been much more sharply defined in the submission. The authors do seem to show that extracting events and event relations can detect a conspiracy news source in the LOCO dataset. But the work could have been richer and more convincing if the authors used their model to explore the kinds of event relations which may mark conspiracy theories, and the ways in which these patterns can inform our understanding of what constitutes a conspiracy theory. For instance, the authors might provide an exploratory analysis of the kinds of spurious event relations that are common to conspiracy theories.

Reviewer dygx also asked an interesting question surrounding soundness. They hypothesized that there might be high-level stylistic differences between conspiracy and non-conspiracy sources which might undermine the findings from the experiments in this paper. If all conspiracy and non-conspiracy sources share stylistic similarities (e.g., if each source is drawn hierarchically from a parent conspiracy or non-conspiracy distribution) then source-based splitting might not be sufficient to check that models are not just learning broad stylistic differences between kinds of news sources. However, in their rebuttal, the authors noted that including event information improved F1 scores in the random split setting (i.e., naively splitting the dataset without considering source). In this setting, the authors argue, event and baseline models each had access to the same stylistic information. But including event relations lead to a 4.22 point increase in F1 score over a vanilla longformer model (80.46 => 84.68). Reviewer dygx did not respond to this argument.

Finally, because of these concerns about stylistic information, the reviewer dygx also noted that they are “not convinced that this model would perform so well with completely new topics.” The authors did attempt to address this comment from dygx in their rebuttal. But their experiments on “topic splitting” were hard to assess because they were lacking detail. The authors do not appear to describe “topic features” in their paper so it is hard to understand this new experiment.
Decision: Accept-Findings