Venue: NAACL 2022

# Early Rumor Detection Using Neural Hawkes Process with a New Benchmark Dataset

Fengzhu Zeng

Singapore Management University 81 Victoria St, Singapore 188065 fzzeng.2020@phdcs.smu.edu.sg Wei Gao

Singapore Management University 81 Victoria St, Singapore 188065 weigao@smu.edu.sg

###### Abstract

Little attention has been paid on EArly Rumor Detection (EARD), and EARD performance was evaluated inappropriately on a few datasets where the actual early-stage information is largely missing. To reverse such situation, we construct BEARD, a new Benchmark dataset for EARD, based on claims from fact-checking websites by trying to gather as many early relevant posts as possible. We also propose HEARD, a novel model based on neural Hawkes process for EARD, which can guide a generic rumor detection model to make timely, accurate and stable predictions. Experiments show that HEARD achieves effective EARD performance on two commonly used general rumor detection datasets and our BEARD dataset.

+
Footnote †: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4105 - 4117


ering the earliness of relevant posts to gather.

Meanwhile, EARD methods have not been well studied either in the literature. Prior works claimed as being able to do early detection can be divided into two categories, both of which are sub-optimal: 1) Methods that are unable to automatically determine a time point for confirming the detection (Zhao et al., 2015; Nguyen et al., 2017; Wu et al., 2017; Liu and Wu, 2018; Xia et al., 2020). Typically, such methods apply a generic rumor detection model to report a decision of classification (e.g., rumor or non-rumor) at each of the _pre-determined_ checkpoints while leaving the determination of the best detection point to human judge. This is subject to a delayed decision as the results at the later checkpoints also need to be examined. 2) Methods that are trained to automatically determine an early detection point, but cannot guarantee the stability of decision (Zhou et al., 2019; Song et al., 2019). For example, CED (Song et al., 2019) decides an early detection point using a fixed probability threshold to assess if the current prediction is credible or not. However, prediction probability does not really reflect model's confidence (Guo et al., 2017), and such decision without properly modeling the _uncertainty beyond the decision point_ may fail to give a timely and reliable detection because the prediction could flip over and over again afterwards with new posts flow in.

In this work, we propose a new method called Hawkes EArly Rumor Detection (HEARD) to model the stabilization process of rumor detection based on a Neural Hawkes Process (NHP) (Mei and Eisner, 2017), which can automatically determine when to make a timely and stable decision of detection. The basic idea is to construct a detection stability distribution over the _expected future_ predictions based on a sequence of _prior and current_ predictions, such that an optimal time point can be fixed without any delay for awaiting and checking the upcoming data beyond that point. Our main contributions can be summarized as follows1: Footnote 1: Dataset and source code are released at [https://github.com/znhy1024/HEAD](https://github.com/znhy1024/HEAD)

* We introduce BEARD, the first EARD-oriented dataset, collected by covering as much as possible the early-stage information relevant to the concerned claims.
* We propose HEARD, a novel EARD model based on the NHP to automatically determine an optimal time point for the stable decision of early detection.
* Extensive experiments show that HEARD achieves more effective EARD performance as compared to strong baselines on BEARD and two commonly used general rumor detection datasets.

## 2 Related Work

### Early Rumor Detection

Despite extensive research on general rumor detection, early detection has not been studied well. Many studies claimed that their general detection models can be applied to early detection by simply fed with data observed up to a set of pre-determined checkpoints (Ma et al., 2016; Yu et al., 2017; Ma et al., 2017, 2018; Guo et al., 2018; Bian et al., 2020). Nevertheless, how to determine an optimal early detection point from many checkpoints is missing and non-trivial, as deciding when to stop often needs to check the data or model's outputs after the current checkpoint, causing delays of detection.

Some methods were claimed further as designed for early detection. Zhao et al. (2015b) proposed to gather related posts with skeptical phrases, and performed detection with cluster-based classifiers over real-time posts. Nguyen et al. (2017) developed a hybrid neural model for post-level representation and credit classification, which were incorporated with the temporal variations of handcrafted features for detecting rumors. Wu et al. (2017) clustered relevant posts and selected key features from clusters to train a topic-independent classifier for revealing emergent rumors. Xia et al. (2020) employed burst detection to segment an event into sub-events and trained an encoder for each sub-event representation for incremental prediction. None of the above methods really address the key issues of early detection as they lack mechanisms enforcing the earliestness, and they cannot automatically fix an optimal detection point either.

ERD (Zhou et al., 2019) used deep reinforcement learning to enforce model to focus on early time intervals for the trade-off between accuracy and earlierness of detection, and is the first EARD method that can automatically decide to stop or continue at a checkpoint. Song et al. (2019) proposed another EARD method called Credible Detection Point 

(CED) using a fixed probability threshold to determine if detection process should stop depending on the credibility of current prediction. However, these models are unstable or of low confidence because the uncertainty of future predictions is not taken into account in training.

### Rumor Detection Datasets

Quite a few rumor detection datasets based on social media posts relevant to a set of claims were released, such as TWITTER Ma et al. (2016), PHEME Zubiaga et al. (2016), RumourEval-2017/19 Derczynski et al. (2017); Gorrell et al. (2019), FakeNewsNet Shu et al. (2020), etc.. RumourEval-2017/19 are minor variants of PHEME while FakeNewsNet is never used for EARD. These datasets were built for general detection of rumors without much consideration on the earliness of information. Thus, the actual early-stage social engagements may not be covered by different data collection mechanisms used, such as applying a rigid time cut-off in Search API or launching a real-time gathering with Streaming API _after_ a news outbreak. To our best knowledge, there is no dataset specifically built for early rumor detection task.

## 3 BEARD Corpus Construction

We scrape the text of title, claim, debunking time and veracity label in the articles on the fact-checking website snopes.com. Our goals are two-fold: 1) The collected posts are not only relevant to the claim but can diversely cover copious variations of relevant text expressions; 2) The collection can cover posts of early arrival, possibly ahead of the pertinent news exposure on the mainstream media.

To this end, we firstly construct high-quality search queries for Twitter search. An original query is formed from the title and claim of each article, with stop words removed. Since the lengthy query might harm the diversity of search results, we utilize some heuristics to obtain a substantial set of variants of each query potentially with better result coverage in Twitter search: i) We preform synonym replacement to create a set of variants of the query; ii) We shorten each variant by removing its words one by one with carefully crafted rules to maintain useful information, e.g., named entities, for good search quality, while keeping the remaining words after each removal as a new variant. As a result, we obtain a substantial set of variants of the original query and merge the Twitter search results of each query and all its variants.

To cover early posts, each Twitter search is performed in an iterative fashion. To avoid ground-truth leakage, we first obtain the possible earliest official debunking time of the given claim by cross-checking its similar claims in a range of fact-checking websites (see Appendix A.1). From the earliest debunking time, we search backward for the relevant posts within \(M\) days _prior to_ debunking, and then push back further \(N\) days earlier than before in each iteration until the number of newly gathered posts in an iteration becomes less than 1% of the posts obtained from the previous iteration.

Finally, for each retrieved post, we use its conversation ID to find the root post of the conversation it is engaged in. We utilize Sentence-BERT Reimers and Gurevych (2019) to retain those root posts with cosine similarities to the claim being higher than an empirical threshold. Thus far, we have obtained a set of conversation IDs for each claim which are led by different root posts (see Appendix A.4 for post-processing). Then we fetch from Twitter all the posts in the detected conversations along with the root posts into our final collection as an instance, and label the conversation as rumor if the corresponding claim is from the "Fact Checks" category on the Snopes or non-rumor if it is from the "News" category.

Due to space limit, we provide the details of search queries construction in Appendix A.2, and the settings of iterative Twitter search in Appendix A.3.

## 4 Problem Definition

Let \({\bf C}=\left\{C\right\}\) denote a set of instances, where each \(C=\left\{y,S\right\}\) consists of the ground-truth label \(y\in\left\{0,1\right\}\) and a set of relevant posts in chronological order \(S=\left\{({\bf m}_{1},\tau_{1}),...,({\bf m}_{i},\tau_{i}),...,({\bf m}_{| S|},\tau_{|S|})~{}\right..~{}y\) indicates \(C\) is a rumor if \(y=1\) or a non-rumor otherwise. \(|S|\) is the number of relevant posts in \(S\). Each tuple \(({\bf m}_{i},\tau_{i})\in S\) includes the text content \({\bf m}_{i}\) and the timestamp \(\tau_{i}\) of the \(i\)-th post, where \(\tau_{i}\) is defined as the time difference between the first and the \(i\)-th post, such that \(\tau_{1}=0\) and \(\tau_{i-1}\leq\tau_{i}\) for \(i>1\). In other words, \(\tau_{i}\) can be regarded as the elapsed time relative to the earliest post so that the timelines of different instances are aligned 

Following the pre-processing method in most prior studies [16, 20, 34], we divide each posts sequence into a sequence of intervals to avoid excessively long sequence. We chop a sequence \(S\) into intervals based on three strategies: 1) fixed posts number, 2) fixed time length and 3) variable length in each interval [29]. Hence \(S\) is converted to \(X=\{(\mathbf{x}_{i},t_{i})\}_{i=1}^{|X|}\), where \(|X|\) is the number of intervals, \(\mathbf{x}_{i}=\{\mathbf{m}_{i,1},\mathbf{m}_{i,2},...,\mathbf{m}_{i,|\mathbf{ x}_{i}|}\}\) and \(t_{i}=\tau_{i,|\mathbf{x}_{i}|}\) which is the timestamp of the _last_ post \(\mathbf{m}_{i,|\mathbf{x}_{i}|}\) in the \(i\)-th interval. Then, we merge the posts falling into the same interval as a single post.

We define the EARD task as automatically determining the _earliest_ time \(\hat{t}\in\{t_{i}\}\), such that the prediction \(\hat{y}\in\{0,1\}\) at \(\hat{t}\) for a given claim is _accurate and remains unchanged_ afterwards with time goes by. It is worthwhile to mention that since \(t_{i}\) relates to the granularity of intervals, it might affect the precision of a decision point based on the formed intervals. In practice, however, we will try to make the intervals small for keeping such impact marginal.

## 5 HEARD Model

Figure 1 shows the architecture of HEARD, which contains two components: 1) the rumor detection component predicts rumor/non-rumor label at each time step/interval; 2) the stabilization component models the prediction stabilization process and determines when to stop at the earliest detection point. We will describe them with detail in this section.

### Rumor Detection Modeling

A standard LSTM cell [8] followed by a fully-connected layer is utilized for rumor detection in each interval. For any \((\mathbf{x}_{i},t_{i})\in X\), \(\mathbf{x}_{i}\) can be turned into a vector \(\mathbf{e}_{i}\) by a text representation method, e.g., TF-IDF [7], CNN [10], BERT [11], etc.. Taking \(\mathbf{e}_{i}\) as input, the LSTM cell gets the hidden state \(\mathbf{h}_{i}=\text{LSTM}(\mathbf{e}_{i})\) and forwards it through the fully-connected layer to perform prediction. The predicted class probability distribution of an instance at \(t_{i}\) is calculated as \(\mathbf{p}_{i}^{r}=\sigma(\mathbf{W}\mathbf{h}_{i}+\mathbf{b})\) and thus the predicted class is \(\hat{y}_{i}=\text{argmax}(\mathbf{p}_{i}^{r})\), where \(\sigma(\cdot)\), \(\mathbf{W}\) and \(\mathbf{b}\) are sigmoid function, weight matrix and bias, respectively.

### Stabilization Process Modeling

**Prediction Inverse (PI).** Our rumor detection component keeps observing the posts stream and outputs a prediction sequence \(\hat{y}_{1},\hat{y}_{2},\hat{y}_{3},\ldots\) along the time steps. During the process, newly arrived posts may provide updated features rendering the next decision of rumor detection to invert from rumor to non-rumor or the other way round. Presumably, the predictions would get stabilized when sufficient clues are accumulated over time. By modeling such a process, we aim to fix the earliest time \(t_{i}\) when the model can produce a stable prediction, meaning that there will be no expected inverses of prediction

Figure 1: The architecture of HEARD. When the current observation arrives at time \(t_{i}\), rumor detection predictions from LSTM \(\hat{y}_{i}\) and \(\hat{y}_{i-1}\) are used to update PI count \(I(t_{i})\). Then the intensity function \(\lambda(t)\) is computed by CTLSTM for \(t>t_{i}\). HEARD will determine \(t_{i}\) as the earliest time point with stable prediction \(\hat{y}_{i}\) and stop if the stability prediction \(\hat{I}(t_{\infty})\) equals to \(I(t_{i})\), or continue otherwise.

 

occurring from \(t_{i}\) onward. Thus, we need to train the model for learning the expected future PIs to maximize its stability of prediction.

As a future PI can occur unforseeably at any time, we introduce \(I(t)\) as a cumulative count of PIs up to any time point \(t\) in a _continuous_ time space to accommodate the uncertainty. Then a PI counts sequence \(I(t_{1}),I(t_{2}),I(t_{3})\ldots\) can be obtained from the prediction sequence. Clearly, we have \(I(t_{1})=0\), and \(I(t_{i})=I(t_{i-1})+1\) if \(\hat{y}_{i}\neq\hat{y}_{i-1}\) or \(I(t_{i})=I(t_{i-1})\) otherwise. Additionally, we denote \(\mathcal{H}_{t_{i}}=\{(I(t_{1}),t_{1}),\dots,(I(t_{i-1}),t_{i-1})\}\) the history of PI counts up to \(t\in(t_{i-1},t_{i}]\), and denote \(\Delta I(t_{i},t_{j})=I(t_{j})-I(t_{i})\) the difference of PI counts between \(t_{i}\) and \(t_{j}\) for \(j>i\).

**Neural Hawkes Process (NHP).** General Hawkes process Hawkes (1971) is a doubly stochastic point process for modeling sequence of discrete events in continuous time, which has been successfully applied in social media research, such as modeling the popularity of tweets Zhao et al. (2015a), rumor stance classification Lukasik et al. (2016), fake retweeter detection Dutta et al. (2020), and extracting temporal features for fake news detection Murayama et al. (2020). Given a sequence of events \(E_{1},E_{2},E_{3},\ldots\) which occur at the corresponding time points \(t_{1},t_{2},t_{3},\ldots\) and \(N(t)\) that is the number of events occurring up to time \(t\), a univariate Hawkes process with a conditional intensity that indicates expected arrival rate of future events at \(t\) is defined as Yang and Zha (2013):

\[\lambda(t)=\mu+\int_{-\infty}^{t}\kappa(t-s)\mathrm{d}N(s) \tag{1}\]

where \(\mu\geq 0\) is the base intensity of event and \(\kappa(\cdot)\) is a _manually specified monotonic kernel function_ that shows how the excitation from history decays with time. It assumes that arrived events can temporarily raise the probability of future events but the influence monotonically decays over time. However, this assumption is very strong which limits its ability for modeling complex dynamic point processes. In rumor diffusion, prediction inverse is the event influenced by many factors, such as what users express in historical and upcoming posts, which may bring tremendous uncertainty of prediction invalidating the monotonic decay assumption. Thus, we propose to adopt an NHP Mei and Eisner (2017) to capture the complex effects by utilizing a RNN with continuous-time LSTM (CTLSTM) to _learn_ the intensity function. CTLSTM extends the vanilla LSTM with an interpolation-like mechanism so that its hidden state for controlling intensity can be updated _discontinuously_ with each event occurrence and also evolves _continuously_ as time elapses towards the next upcoming event.

**Intensity Function Estimation.** We use NHP to model the dynamics of PI (i.e., event) and approximate a detection stability distribution over the expected future predictions based on the sequence of historical and current predictions. As shown in Figure 1, CTLSTM reads the current observation \((I(t_{i}),t_{i})\) to obtain \(\lambda(t)\) for any \(t>t_{i}\), so that the distribution over the expected PI counts can be approximated. The intensity function \(\lambda(t)\) is controlled by a hidden state \(\widetilde{\mathbf{h}}(t)\) as follow:

\[\lambda(t)=f(\widetilde{\mathbf{W}}\cdot\widetilde{\mathbf{h}}(t)) \tag{2}\]

where \(\widetilde{\mathbf{W}}\) is a weight matrix and \(f(x)=\beta\log(1+\exp(x/\beta))\) is the softplus function to obtain a positive intensity with a scale parameter \(\beta\)Mei and Eisner (2017).

To model the unknown future for \(t>t_{i}\) based on historical representation \(\widetilde{\mathbf{h}}(t_{i})\), a new hidden cell vector \(\widetilde{\mathbf{c}}(t)\) is introduced to control how \(\widetilde{\mathbf{h}}(t)\)_continuously_ evolves over time. Specifically, \(I(t_{i})\) is firstly transformed to a vector at \(t_{i}\) by a fully-connected layer to join the updates in CTLSTM. Then, CTLSTM updates the hidden state \(\widetilde{\mathbf{h}}(t)\) that has been evolving towards \(\widetilde{\mathbf{h}}(t_{i})\) based on \(\widetilde{\mathbf{c}}(t)\) for \(t\in(t_{i-1},t_{i}]\). Thus, a richer representation of history can be learned by taking into account the dynamics of impact between the two consecutive observations. Meanwhile, to model the expected future PI count, \(\widetilde{\mathbf{c}}(t)\) is updated to a new state with the current cell input, which is analogous to how vanilla LSTM updates hidden cell2, and from the new state, \(\widetilde{\mathbf{c}}(t)\) begins to continually approximate a target state that is defined by CTLSTM to represent an expected state of \(\widetilde{\mathbf{c}}(t)\) for \(t\rightarrow\infty\).

Footnote 2: The difference is that vanilla LSTM updates the hidden cell based on that of the previous time step while the update here is based on \(\widetilde{\mathbf{c}}(t)\) for \(t\to t_{i}\).

**Expected Stabilization.**\(\lambda(t)\) indicates the expected instantaneous rate of future PIs from \(t\) onward. We can predict the value of \(I(t_{\infty})\) (\(t_{\infty}\) denotes \(t\rightarrow\infty\)) and further determine an expected earliest stabilized observation \((I(t_{i^{*}}),t_{i^{*}})\), such that \(t_{i^{*}}=\text{min}\{t_{i}\mid\Delta I(t_{i},t_{\infty})=0,i\in\{1,\dots,|X|\}\}\). Hence, \(t_{i^{*}}\) 

earliest time that the predictions remain unchanged after it. We approximate \(\Delta I(t_{i},t_{\infty})\) by

\[\Delta\hat{I}(t_{i},t_{\infty})=\int^{\infty}_{t_{i}}\lambda(t)\mathrm{d}t \tag{3}\]

where \(\hat{I}(\cdot)\) constantly denotes a predicted PI count. We use Monte Carlo trick to handle all integral estimations Mei and Eisner (2017). Given \(t_{i^{*}}\), HEARD finally outputs \(\hat{y}_{i^{*}}\) as the stable prediction for early rumor detection.

### HEARD Model Training

We utilize a next observation \((I(t_{i+1}),t_{i+1})\) prediction task to train CTLSTM. With PIs as the indicator of model stability, CTLSTM aims to fit the sequence of observations \(\mathcal{H}_{t_{i+1}}\) obtained from the predictions of the rumor detection module.

Specifically, \(\hat{I}(t_{i+1})\) can be obtained from \(I(t_{i})+\Delta\hat{I}(t_{i},t_{i+1})\), and the difference value \(\Delta\hat{I}(t_{i},t_{i+1})\) is either 1 or 0 which can be predicted as \(\Delta\hat{I}(t_{i},t_{i+1})=\text{argmax}(\mathbf{p}^{c}_{i})\), where we infer \(\mathbf{p}^{c}_{i}=\sigma(\mathbf{W}^{\prime}\widehat{\mathbf{h}}(t_{i})+ \mathbf{b}^{\prime})\) with trainable parameters \(\mathbf{W}^{\prime}\) and \(\mathbf{b}^{\prime}\). For predicting \(t_{i+1}\), a density \(p_{i}(t)\) is formulated as

\[p_{i}(t)=\lambda(t)\exp\left(-\int^{t}_{t_{i}}\lambda(s)\mathrm{d}s\right) \tag{4}\]

Then we use the minimum Bayes risk predictor for time prediction Mei and Eisner (2017):

\[\hat{t}_{i+1}=\mathbb{E}[t_{i+1}|t_{i},\mathcal{H}_{t_{i}}]=\int^{\infty}_{t_ {i}}p_{i}(t)t\mathrm{d}t \tag{5}\]

where \(\mathbb{E}[\cdot]\) is an estimator for choosing an optimal time point \(t\) to minimize the expectation of risks.

The overall loss consists of three terms on rumor detection, expected earliest stable time and CTLSTM. Concretely, given an instance \(C\) with input sequence \(X=\{(\mathbf{x}_{i},t_{i})\}\), let \(\mathbf{y}\) be the one-hot encoding of ground-truth label (i.e., rumor or not). At each time of observation \(t_{i}\), the cross-entropy loss between prediction and ground truth is defined as \(L^{r}_{i}=-\mathbf{y}\cdot\log\mathbf{p}^{r}_{i}\). For the expected earliest stable time, the loss at \(t_{i}\) is defined as

\[L^{e}_{i}=|\Delta\hat{I}-\Delta I|-\log\left(1-\frac{i}{|X|}\right) \tag{6}\]

where the first term is the loss of \(\Delta I(t_{i},t_{\infty})\)3 approximation in Eq. 3, and the second term encourages the model to select an earliest time possible. The loss incurred from CTLSTM is given as

Footnote 3: For simplicity, \(\Delta I(t_{i},t_{\infty})\) and \(\Delta\hat{I}(t_{i},t_{\infty})\) are denoted as \(\Delta I\) and \(\Delta\hat{I}\) in Eq. 6, respectively.

\[L^{c}_{i}=-\Delta\mathbf{I}\cdot\log\mathbf{p}^{c}_{i}+|\hat{t}_{i+1}-t_{i+1}| \tag{7}\]

where \(\Delta\mathbf{I}\) is the one-hot encoding of target \(\Delta I(t_{i},t_{i+1})\), the first term is the loss of next prediction inverse and the second term is the loss of time prediction for next observation.

Our objective is to minimize the cumulative loss up to \(t_{i^{*}}\) when the early detection decision is made: \(L=\frac{1}{i^{*}}\sum_{i=1}^{i^{*}}L_{i}\), where \(L_{i}=L^{r}_{i}+L^{e}_{i}+L^{c}_{i}\). We use stochastic gradient decent (SGD) mini-batch training over all training instances.

## 6 Experiments and Results

### Experimental Setup

**Datasets.** We use BEARD, TWITTER Ma et al. (2016) and PHEME Zubiaga et al. (2016) datasets in the evaluation. BEARD contains 1,198 rumors and non-rumors reported during 2015/03-2021/01 with around 3.3 million relevant posts. We hold out 20% of instances for tuning, and the rest are randomly split with a ratio of 3:1 for training/test. Results are averaged over 5 splits. In Table 2, we show the statistics of TWITTER, PHEME and BEARD datasets.

**Baselines.** We compare HEARD with four state-of-the-art baselines using their original source codes: 1) **BERT**Devlin et al. (2019) is fine-tuned on the "earliest rumor detection" task Miao et al. (2021), in which the early detection strategy is to output a prediction using only the _first_ post of each instance. 2) **CED**Song et al. (2019) uses a fixed probability threshold to check if the prediction is credible for determining the early detection point. 3) **ERD**Zhou et al. (2019) uses a Deep Q-Network DQN to enforce the model to focus on early posts for determining the time point to stop and output the detection result. 4) **STN**Xia et al. (2020) use a time-evolving network to represent

\begin{table}
\begin{tabular}{l|l|l|l|l} \hline \hline Dataset & \multicolumn{2}{l|}{Instances \#} & Posts \# & AvgLen (hrs) \\ \hline TWITTER & R & 498 & 182,499 & 2,538 \\  & N & 494 & 466,480 & 1,456 \\ \hline \multirow{2}{*}{PHEME} & R & 1,972 & 31,230 & 10 \\  & N & 3,830 & 71,210 & 19 \\ \hline \multirow{2}{*}{BEARD} & R & 531 & 2,644,807 & 1,432 \\  & N & 667 & 657,925 & 1,683 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Statistics of datasets. R: Rumor; N: Non-rumor.

 

state-independent sub-events in posts sequence for classifying claims at each checkpoint.

### Experimental Settings

To balance the sequence length and granularity of time intervals, we pre-process posts sequences in the three datasets differently. We merge every 10 posts in BEARD and every 2 posts in TWITTER while we only merge the posts with the same timestamp in PHEME due to its generally short sequences. For each interval, both ERD and STN use pre-trained word embeddings to initialize the embedding matrix and fine-tune it in the training, while CED uses the TF-IDF method Salton and Buckley (1988), all of which follow the settings in the original papers Song et al. (2019); Zhou et al. (2019); Xia et al. (2020). We also follow the original CED setting by using TF-IDF with 1,000 dimensions for representing the posts in each time interval.

The hidden size of standard LSTM is set to 128 with the dropout rate of 0.1, and the size of CTL-STM is set to 64. We pad all the sequences in a batch to the same length as the longest one, with the batch size of 16. We use the Adam Kingma and Ba (2015) with a learning rate of \(2e\)-4 for optimization. To avoid overfitting, we add a L2 regularization with the weight of 1e-4. All values are fixed based on the validation set.

Our model HEARD is implemented using Pytorch4. We use the original source codes of all the baselines: CED5 and ERD6 are implemented with TensorFlow; BERT7 are implemented with PyTorch, and we use the base uncased pre-trained model; The code of STN is obtained directly from the authors of the original paper Xia et al. (2020) which is implemented with Pytorch. All the experiments are conducted on a server with 4*12GB NVIDIA GeForce RTX 2080 Ti GPUs.

Footnote 4: [https://pytorch.org/](https://pytorch.org/)

Footnote 5: [https://github.com/thunlp/CED](https://github.com/thunlp/CED)

Footnote 6: [https://github.com/DeepBrainAI/ERD](https://github.com/DeepBrainAI/ERD)

Footnote 7: [https://github.com/huggingface/transformers](https://github.com/huggingface/transformers)

### Evaluation Metrics

We use the general classification evaluation metrics accuracy and F1-score together with several EARD-specific metrics (see below) for evaluation.

**Early Rate (ER)**Song et al. (2019) is defined as the utilization ratio of posts: \(ER=\frac{1}{|\mathbf{C}|}\sum_{C\in\mathbf{C}}\frac{i_{C}}{|C|}\) where \(\mathbf{C}\) is the test set, \(i_{C}\) implies the early detection decision is made at the \(i\)-th post in instance \(C\) and \(|C|\) is the number of posts in it. _Lower_ ER means the model can detect rumors _earlier_.

**Early Detection Accuracy Over Time (EDAOT).** The metric of detection accuracy over time widely used Ma et al. (2016); Zhou et al. (2019); Xia et al. (2020) is unsuitable for EARD models as it enforces a model to output a decision at each checkpoint whereas an EARD model can decide its own optimal decision point which may be earlier and more accurate than its output at the checkpoint. Our variant requires a model output result _only_ when it cannot make an early decision before a given checkpoint while both accuracy and average time of decisions will be presented. Specifically, given a set of checkpoints at time \(\{t_{1},...,t_{k}\}\), at the \(j\)-th checkpoint, the detection accuracy is \(Acc_{t_{j}}=\sum_{C\in\mathbf{C}}\frac{\mathds{I}_{\hat{y}_{i^{*}}=y}}{| \mathbf{C}|}\) , where the binary function \(\mathds{1}\) takes 1 if \(\hat{y}_{i^{*}}=y\) or 0 otherwise. And the average time of decisions is \(AvgT_{t_{j}}=\sum_{C\in\mathbf{C}}\frac{t_{i^{*}}}{|\mathbf{C}|}\), where \(t_{i^{*}}\leq t_{j}\), and \(t_{i^{*}}=t_{j}\) if the model cannot make a decision before \(t_{j}\).

**Stabilized Early Accuracy (SEA)** is a newly defined comprehensive metric considering accuracy, earliness and stabilization:

\[SEA =\frac{1}{3|\mathbf{C}|}\sum_{C\in\mathbf{C}}\left[\mathds{1}_{ \hat{y}_{i^{*}}=y}+\left(1-\frac{i_{C}}{|C|}\right)\right.\] \[~{}~{}~{}~{}+\left.\left(1-\frac{\Delta I(t_{i^{*}},t_{|X|})}{|X|- i^{*}}\right)\right]\]

where the first term is the ratio of correctly predicted instances at the predicted time point \(t_{i^{*}}\) indicating accuracy, the second term is the ratio of posts after \(t_{i^{*}}\) indicating earliness, and the third term is the ratio of unchanged predictions after \(t_{i^{*}}\) indicating stability. The value of SEA is bounded in \([0,1]\) and higher SEA means better performance.

### Results and Analysis

The main results are provided in Table 3. We show the EDAOAT results in Figure 2.

**Results of Classification.** STN uses the entire timeline of each instance since it cannot automatically determine an early detection point. As shown in Table 3, however, it only achieves comparable Accuracy and F1 as ERD and CED and is much worse than HEARD, even though it was reported much better than CED on TWITTER in previous 

work (Xia et al., 2020)8. This implies that the early detection models are promising which can use a prior fraction of posts to achieve similar or much better results. It also suggests that capturing early-stage features is important to more accurate rumor detection. Only using the source post for detection, BERT gives worst Accuracy and F1 on TWITTER and BEARD, but it gets unexpectedly high performance on PHEME. We inspect this issue by following the prior analysis (Schuster et al., 2019) based on Local Mutual Information (LMI) (Evert, 2005) and Pointwise Mutual Information (PMI) (Church and Hanks, 1990). We find that the source posts in PHEME have spuriously much stronger correlation with the class labels. Appendix B.1 discusses such bias and the possible cause. This observation suggests data sampling bias exists in the existing dataset, and thus the model's decision based on the source post only can be misleading and insufficient.

Footnote 7: The CED performance reported in (Xía et al., 2020) is an excerpt from (Song et al., 2019) based on half of the TWITTER data, while they experimented STN using the full data.

**Results of ER and SEA.** Table 3 also shows that HEARD consistently outperforms ERD and CED in large margin based on ER and SEA, indicating HEARD is more effective and stable. HEARD considerably improves CFD by \(57\%\), \(68\%\) and \(27\%\) in ER and by \(27\%\), \(32\%\) and \(11\%\) in SEA on TWITTER, PHEME and BEARD, respectively. ERD's high ER scores entails that it can hardly make early decision, as this DQN-based model seems weak on its reward function, which gives a small penalty to continuation but a large one to termination with wrong predictions (Zhou et al., 2019), discouraging it from stopping early. BERT only uses the first post for detection which thus obtains the lowest ER. Note that a model being _expectantly_ stable at the time of decision means the prediction will remain unchanged even though new data could be seen by the model after that. To probe its stability, we enforce model to continue outputting predictions at the checkpoints later than its decision point. We can see that HERAD still outperforms BERT on SEA on all the datasets indicating it is more stable.

**Results of EDAOT.** Figure 2 shows that HEARD is clearly superior over the baselines in accuracy, earliness and stability at the checkpoints. ERD looks relatively stable but it hardly makes early detection due to aforementioned reason. Our conjecture is that its DQN module forces it to overly focus on the first few intervals while deferring the decision to the end due to the weak reward design, resulting in nearly no improvement even with more data. Note that in EDAOT evaluation a model should stop once it outputs a prediction for a given checkpoint (i.e., deadline). Thus, BERT reports all the predictions using only the first post, rendering the same accuracy at checkpoints. Interestingly, HEARD can make especially fast decisions on PHEME which only uses a little less than 6 hours given a 48-hour deadline. The reason might be the average length

Figure 2: Early detection accuracy over time within 48 hours. X-axis: timeline; Y-axis: accuracy; Vertical lines: checkpoints; Dots with shapes: decision points; Time above a dot: Deadline (i.e., checkpoint) set for the model.

 

of instances in PHEME is only around 14 hours as compared to over 1,000 hours in other two datasets, which HEARD can especially benefit from in the training. However, considering the fact that the PHEME (and TWITTER) dataset may fail to cover the real early information, such a very quick detection on it could be an illusion since the patterns the model actually uses for the decision are from the midst of rumor propagation.

## 7 Case Study

In this section, we give an intuitive illustration to reveal 1) why BEARD is more suitable for EARD task; and 2) how HEARD is more advantageous than the state-of-the-arts with automatic early time point determination. Specifically, we recollect the relevant posts of rumor cases in PHEME and TWITTER using our data construction method for BEARD. To analyze the first question, we inspect the posts before and after the recollection as shown in Figure 3. In the original posts, the actual early-stage timelines of both cases are largely missing. Our recollected data can cover the important denial and questioning posts signaling rumors in the early stage. This observation indicates that our data collection method has improved coverage by recalling these actual early information which is not available in the existing datasets.

To analyze the second question, we display the detection outputs of different models before and after the recollection. HEARD consistently detects rumors with less time than CED and ERD since it automatically terminates when stabilization process estimates the expected future prediction inverse will not change. The correctness of such expectation could be reflected by the value of semantic similarity, which almost has no change with new posts coming in after the detection point, implying the future prediction results are expected stable.

## 8 Conclusion and Limitation

We introduce BEARD, a new benchmark dataset collected for early rumor detection, and propose a model called HEARD to perform stable early rumor detection based on neural Hawkes process. Experiments show that HEARD achieves overall better performance than state-of-the-art baselines. Analysis entails BEARD is more suitable for the early rumor detection task.

There are some cases that a rumor seems to be stable over an extended time period but is eventually refuted by an authoritative source later. It is because sensational discussions are more attractive to some social media users than facts, which leads to signals in support of the rumor are usually much stronger in social media, and the final correction often has only small engagement. In such cases, our model HEARD based on social media alone could be misled by the signals of only one source. In future, we plan to study incorporating authoritative sources of information to alleviate this phenomenon on social media, such as statements of official platforms, scientific sources, etc..

## Acknowledgement

This research is supported by the National Research Foundation, Singapore under its Strategic Capabilities Research Centres Funding Initiative and the

Figure 3: (a) A rumor in PHEME on “The co-pilot of Germanwings flight 9525 was Muslim convert”; (b) A rumor in TWITTER on “A photograph shows the darkest baby in the world”. Y-axis: the number of collected posts; X-axis: timeline in minutes (m)/days (d); Colored bar: the semantic similarity between the posts in any time step and the posts in the decision time step, where the red bar denotes recollected data and the blue one denotes original data. The similarity ranges from 0.7 to 1.0, and the darker the color the more similar the posts are.

 

Singapore Ministry of Education (MOE) Academic Research Fund (AcRF) Tier 1 grant. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of funding agencies.

## References

* Bian et al. (2020) Tian Bian, Xi Xiao, Tingyang Xu, Peilin Zhao, Wenbing Huang, Yu Rong, and Junzhou Huang. 2020. Rumor detection on social media with bi-directional graph convolutional networks. In _Proceedings of the 34th AAAI conference on artificial intelligence_, pages 549-556.
* Bird et al. (2009) Steven Bird, Ewan Klein, and Edward Loper. 2009. _Natural language processing with Python: analyzing text with the natural language toolkit_. O'Reilly Media, Inc.
* Church and Hanks (1990) Kenneth Ward Church and Patrick Hanks. 1990. Word association norms, mutual information, and lexicography. _Computational Linguistics_, 16(1):22-29.
* Derczynski et al. (2017) Leon Derczynski, Kalina Bontheva, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Arkaitz Zubiga. 2017. SemEval-2017 task 8: RumourEval: Determining rumour veracity and support for rumours. In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_, pages 69-76, Vancouver, Canada. Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1__Long and Short Papers_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Dutta et al. (2020) Hridoy Sankar Dutta, Vishal Raj Dutta, Aditya Abhikary, and Tanmoy Chakraborty. 2020. Hawkeseye: Detecting fake retweeters using hawkes process and topic modeling. _IEEE Transactions on Information Forensics and Security_, 15:2667-2678.
* Evert (2005) Stefan Evert. 2005. _The statistics of word cooccurrences: word pairs and collocations_. Ph.D. thesis, University of Stuttgart.
* Gorrell et al. (2019) Genevieve Gorrell, Elena Kochkina, Maria Liakata, Ahmet Aker, Arkatiz Zubiaga, Kalina Bontheeva, and Leon Derczynski. 2019. SemEval-2019 task 7: RumourEval, determining rumour veracity and support for rumours. In _Proceedings of the 13th International Workshop on Semantic Evaluation_, pages 845-854, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
* Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In _Proceedings of the 34th International Conference on Machine Learning_, volume 70, pages 1321-1330.
* Guo et al. (2018) Han Guo, Juan Cao, Yazi Zhang, Junbo Guo, and Jintao Li. 2018. Rumor detection with hierarchical social attention network. In _Proceedings of the 27th ACM International Conference on Information and Knowledge Management_, pages 943-951.
* Hawkes (1971) Alan G Hawkes. 1971. Spectra of some self-exciting and mutually exciting point processes. _Biometrika_, 58(1):83-90.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_, 9(8):1735-1780.
* Kim (2014) Yoon Kim. 2014. Convolutional neural networks for sentence classification. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1746-1751, Doha, Qatar. Association for Computational Linguistics.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _Proceedings of the 3rd International Conference on Learning Representations_.
* Liu and Wu (2018) Yang Liu and Yi-fang Brook Wu. 2018. Early detection of fake news on social media through propagation path classification with recurrent and convolutional networks. In _Proceedings of the 32nd AAAI conference on artificial intelligence_, pages 354-361.
* Lukasik et al. (2016) Michal Lukasik, P. K. Srijith, Duy Vu, Kalina Bontheva, Arkaitz Zubiqaa, and Trevor Cohn. 2016. Hawkes processes for continuous time sequence classification: an application to rumour stance classification in Twitter. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 393-398, Berlin, Germany. Association for Computational Linguistics.
* Ma et al. (2016) Jing Ma, Wei Gao, Prasenjit Mitra, Sejeong Kwon, Bernard J. Jansen, Kam-Fai Wong, and Meeyoung Cha. 2016. Detecting rumors from microblogs with recurrent neural networks. In _Proceedings of the 25th International Joint Conference on Artificial Intelligence_, pages 3818-3824.
* Ma et al. (2017) Jing Ma, Wei Gao, and Kam-Fai Wong. 2017. Detect rumors in microblog posts using propagation structure via kernel learning. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 708-717, Vancouver, Canada. Association for Computational Linguistics.
* Ma et al. (2018) Jing Ma, Wei Gao, and Kam-Fai Wong. 2018. Rumor detection on Twitter with tree-structured recursive neural networks. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1980-1989, Melbourne, Australia. Association for Computational Linguistics.
* Mei and Eisener (2017) Hongyuan Mei and Jason Eisener. 2017. The neural hawkes process: A neurally self-modulating multivariate point process. In _Advances in Neural Information Processing Systems 30_, pages 6754-6764.
* Miao et al. (2021) Xin Miao, Dongning Rao, and Zhihua Jiang. 2021. Syntax and sentiment enhanced bert for earliest rumor de 

tection. In _CCF International Conference on Natural Language Processing and Chinese Computing_, pages 570-582.
* Murayama et al. (2020) Taichi Murayama, Shoko Wakamiya, and Eiji Aramaki. 2020. Fake news detection using temporal features extracted via point process. In _Workshop Proceedings of the 14th International AAAI Conference on Web and Social Media_.
* Nguyen et al. (2017) Tu Ngoc Nguyen, Cheng Li, and Claudia Niederree. 2017. On early-stage debunking rumors on twitter: Leveraging the wisdom of weak learners. In _International Conference on Social Informatics_, pages 141-158.
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
* Ruchansky et al. (2017) Natali Ruchansky, Sungyong Seo, and Yan Liu. 2017. CSI: A hybrid deep model for fake news detection. In _Proceedings of the 26th ACM International Conference on Information and Knowledge Management_, pages 797-806.
* Salton and Buckley (1988) Gerard Salton and Christopher Buckley. 1988. Term-weighting approaches in automatic text retrieval. _Information processing & management_, 24(5):513-523.
* Schuster et al. (2019) Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. 2019. Towards debiasing fact verification models. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3419-3425, Hong Kong, China. Association for Computational Linguistics.
* Shu et al. (2020) Kai Shu, Deepak Mahudeswaran, Suhang Wang, Dongwon Lee, and Huan Liu. 2020. Fakenewsnet: A data repository with news content, social context, and spatiotemporal information for studying fake news on social media. _Big data_, 8(3):171-188.
* Song et al. (2019) Changhe Song, Cheng Yang, Huimin Chen, Cunchao Tu, Zhiyuan Liu, and Maosong Sun. 2019. Ced: Credible early detection of social media rumors. _IEEE Transactions on Knowledge and Data Engineering_, 33(8):3035-3047.
* Wu et al. (2017) Liang Wu, Jundong Li, Xia Hu, and Huan Liu. 2017. Gleaning wisdom from the past: Early detection of emerging rumors in social media. In _Proceedings of the 2017 SIAM International Conference on Data Mining_, pages 99-107.
* Xia et al. (2020) Rui Xia, Kaizhou Xuan, and Jianfei Yu. 2020. A state-independent and time-evolving network for early rumor detection in social media. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 9042-9051, Online. Association for Computational Linguistics.
* Yang and Zha (2013) Shuang-Hong Yang and Hongyuan Zha. 2013. Mixture of mutually exciting processes for viral diffusion. In _Proceedings of the 30th International Conference on Machine Learning_, volume 28, pages 1-9.
* Yu et al. (2017) Feng Yu, Qiang Liu, Shu Wu, Liang Wang, and Tieniu Tan. 2017. A convolutional approach for misinformation identification. In _Proceedings of the 26th International Joint Conference on Artificial Intelligence_, pages 3901-3907.
* Zhao et al. (2015a) Qingyuan Zhao, Murat A. Erdogdu, Hera Y. He, Anand Rajaraman, and Jure Leskovec. 2015a. SEISMIC: A self-exciting point process model for predicting tweet popularity. In _Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_, pages 1513-1522.
* Zhao et al. (2015b) Zhe Zhao, Paul Resnick, and Qiazhu Mei. 2015b. Enguring minds: Early detection of rumors in social media from enquiry posts. In _Proceedings of the 24th International Conference on World Wide Web_, pages 1395-1405.
* Zhou et al. (2019) Kaimin Zhou, Chang Shu, Binyang Li, and Jey Han Lau. 2019. Early rumour detection. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 1614-1623, Minneapolis, Minnesota. Association for Computational Linguistics.
* Zubiaga et al. (2016) Arkaitz Zubiaga, Maria Liakata, and Rob Procter. 2016. Learning reporting dynamics during breaking news for rumour detection in social media. _ArXiv preprint_, abs/1610.07363.

## Appendix A Corpus Construction Protocols

### Claims Collection

Snopes9 is a well-known fact-checking website where fact-checkers manually collect check-worthy claims from multiple sources (e.g., social media, e-mail, news, etc.) and review each claim to report a decision in terms of "Fact Checks" (i.e., rumors being fact-checked) or "News" (i.e., non-rumors of no need to check). Further, the checkers verify the rumors and compose detailed fact-checking articles for justifying the veracity of rumors.

Footnote 9: [https://www.snopes.com/](https://www.snopes.com/) 

We only include the claims that were published in recent 6 years since they have relatively more complete exposure on Twitter (e.g., aged posts might be more likely to get deleted); (2) We only include the claims that have relevant posts on Twitter before the claim was officially debunked. As a result, we collect 531 rumor and 667 non-rumor claims reported during 2015/03-2021/01. For each claim, we scrape the text of article title, the claim, the debunking time and the veracity label.

We also try to exclude posts that may leak the ground truth since a claim might have been fact-checked by other fact-checking websites, thus the truth might have been referenced in the social media posts, such as the rumor example in TWITTER shown in Table 1. To minimize the chance of ground truth leakage, therefore, we match each claim from Snopes across the claims from a handful set of fact-checking websites including FactCheck.org10 and PolitiFact.com11 to get the possible earliest official debunking time, from which we begin collecting the relevant posts backwards in time.

Footnote 10: [https://www.factcheck.org/](https://www.factcheck.org/)

Footnote 11: [https://www.politifact.com/](https://www.politifact.com/)

### Query Construction

For each claim, we construct a set of high-quality search queries for Twitter search to diversely cover relevant posts. Firstly, we concatenate the title and claim of an article followed by stop words removal to filter out noise, resulting in an original query. Since the original query might be long and harm the diversity of search results, we carry out the following ad hoc operations to shorten the query for maintaining maximum useful information and possibly broadening the coverage of retrieved posts: (1) We perform synonym replacement with Natural Language Toolkit (NLTK) Bird et al. (2009) to create a variant of the query for each replacement; (2) For each variant obtained in (1), we use Google Search API12 to search for this altered query and rank by the frequencies of the _highlighted words_ that are hit in the top-100 searched snippets; (3) We then shorten the original query and its variants obtained in (2) by removing the highlighted words that are contained in the queries one by one starting from the low-frequency words, while keeping the remaining words after each removal as a variant of the query, until the shortest variant is left with three words. Note that here we perform named entity recognition by NLTK on the original query and the words that are parts of named entities will not be removed as they are useful for search.

Footnote 12: [https://developers.google.com/custom-search/v1/overview](https://developers.google.com/custom-search/v1/overview)

### Iterative Twitter Search

To prevent early quit of iteration caused by the intermittent sparse distribution of posts along the timeline, we manually adjust the values of \(M\) and \(N\) by trial and error with different instances to gather as much early posts as we can. We also manually check the search results based on a sample of instances using different settings of the termination threshold, i.e., the ratio of the number of gathered posts in each iteration over that in the previous iteration. We observe that too high threshold hinders early posts to be searched out, but too low threshold tends to introduce more noise. We finally set 1% as the termination threshold by trading off earliness and noise.

### Posts Collection

Rather than merging the conversations led by different root posts regarding the same claim into a large conversation, it is more realistic to remain them naturally separated since the conversations originate from different sources and the merge may introduce unnecessary bias. And we drop all the posts that are published after the official debunking time of each claim to avoid ground truth leakage.

## Appendix B Experimental Details

### Bias Evaluation on Datasets

As mentioned in Section 6.4, we utilize LMI and PMI to examine the data sampling bias in the existing datasets and BEARD. The top-5 LMI-ranked words of PHEME, TWITTER and BEARD are shown in Table 4. The top-5 LMI-ranked words of PHEME have much higher LMI and PMI than the other two datasets indicating the high correlations between the words of source posts and the label. Meanwhile, the words in PHEME, e.g., 'breaking', 'hostages', 'shot', etc., are more eyes-catching comparing to the top ranked words in other two datasets. These idiosyncrasies might be introduced by the construction method of PHEME in a sense that journalists might see a timeline of posts about the breaking news and then annotate source posts of conversations, which are easily utilized by BERT to obtain high classification performance. Some 

salient bias also exists in TWITTER dataset evidenced as some top words with high LMI and PMI, such as 'black (lives)', 'flag' and 'trump', while we do not observe such bias among the top words in BEARD.

\begin{table}
\begin{tabular}{l|l|c|l} \hline \hline Dataset & Word & LMI\(10^{-6}\) & PMI \\ \hline  & breaking & 2,720 & 0.75 \\ PHEME & hostages & 2,569 & 0.66 \\ soldier & 2,378 & 1.00 \\ shot & 2,308 & 0.79 \\ cafe & 2,283 & 0.74 \\ \hline  & not & 1,214 & 0.38 \\  & black & 780 & 0.45 \\ TWITT & flag & 745 & 0.49 \\  & trump & 637 & 0.39 \\  & baby & 589 & 0.53 \\ \hline  & he & 1,246 & 0.38 \\ BEARD & they & 722 & 0.26 \\ his & 665 & 0.26 \\ by & 598 & 0.19 \\  & biden & 580 & 0.51 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Top-5 LMI-ranked words of source posts correlated with class label in three datasets.