SlowGuess/NAACL_2022_OCR huggingface dataset:
Fengzhu Zeng
81 Victoria St, Singapore 188065
fzzeng.2020@phdcs.smu.edu.sg
Wei Gao
weigao@smu.edu.sg
Little attention has been paid on EArly Rumor Detection (EARD), and EARD performance was evaluated inappropriately on a few datasets where the actual early-stage information is largely missing. To reverse such situation, we construct BEARD, a new Benchmark dataset for EARD, based on claims from fact-checking websites by trying to gather as many early relevant posts as possible. We also propose HEARD, a novel model based on neural Hawkes process for EARD, which can guide a generic rumor detection model to make timely, accurate and stable predictions. Experiments show that HEARD achieves effective EARD performance on two commonly used general rumor detection datasets and our BEARD dataset.
The proliferation of online rumors has aroused widespread concerns. Many studies have been conducted for automatic rumor detection in social media and achieved high detection accuracy (Ma et al., 2016, 2017; Yu et al., 2017; Ruchansky et al., 2017; Ma et al., 2018; Guo et al., 2018; Bian et al., 2020). However, these generic detection methods lack in-depth modeling of temporality, which can cause tremendous delay in detection given the instantaneous and sporadic nature of rumor propagation.
While a few EArly Rumor Detection (EARD) models have been proposed (Liu and Wu, 2018; Zhou et al., 2019; Song et al., 2019; Xia et al., 2020), they have been designed with oversimplification and evaluated inappropriately using the datasets constructed for generic rumor detection. Widely used rumor detection datasets, such as TWITTER (Ma et al., 2016) and PHEME (Zubiaga et al., 2016), are generally limited in covering relevant posts in the early stage as there was no mechanism ensuring to gather information that is further away from the official debunking time of a rumor. For this reason, the generalizability of EARD cannot be
Time	Post
T.ORG.	2015-07-07	Translucent butterfly - beautiful!
2015-10-01	#Snopes Translucent Butterfly URL
T.REC.	2013-02-08	Ever see a translucent butterfly?
2013-06-24	fake...like bubbles
2014-09-03	...fake, a Worth1000 Photoshop contest entry
2014-10-06	Multiple repeat fakes
P.ORG.	01-08 00:00	... courtesy of Banksy.
01-08 14:39	...it is not by Banksy its by @LucilleClerc
P.REC.	01-07 20:09	Banksy for Charlie...
01-07 23:48	That's not Banksy though, just someone fan page
01-08 00:14	...I don't think that Banksy Insta is official is it?
01-08 08:33	The person (not B.) shared it from another source.
01-08 09:55	Not Banksy btw. It's @LucilleClerc
Table 1: A rumor in TWITTER dataset claiming "A butterfly with translucent wings" and another from PHEME claiming "A street artist Banksy posted an illustration on the Instagram as tribute to Charlie Hebdo". T. (P.) denotes TWITTER (PHEME) dataset, and ORG. (REC.) denotes the original (recollected) set of posts. The earliest post in each set is in Italic.
effectively trained nor be genuinely reflected using a general rumor detection dataset. As an example, we showcase two rumors from TWITTER and PHEME datasets in Table 1. We manually trace Twitter conversations about each claim and recollect as many early posts relevant to it as we could. It is observed that the original posts in both datasets are clearly delayed as compared to our recollected posts. Also, the rumor indicative patterns in the recollected posts unfold differently, where dissenting voices, a common indicator of rumor, appear much earlier and may last for many hours or even years, evolving from vaguely opposing the claim (e.g., 'like bubbles', 'someone') to firmly refuting it with evidence (e.g., 'Photoshop contest'). The original "early" posts in PHEME clearly fail to cover such useful patterns reflecting the early dynamics, and the posts in TWITTER do not cover any early indicative signals before the rumor was officially debunked by Snopes. Given the unavailability of EARD-specific dataset, it is necessary to construct a Benchmark dataset for EARD (BEARD) consid-
Human Language Technologies, pages 4105 - 4117
ering the earliness of relevant posts to gather.
Meanwhile, EARD methods have not been well studied either in the literature. Prior works claimed as being able to do early detection can be divided into two categories, both of which are sub-optimal: 1) Methods that are unable to automatically determine a time point for confirming the detection (Zhao et al., 2015b; Nguyen et al., 2017; Wu et al., 2017; Liu and Wu, 2018; Xia et al., 2020). Typically, such methods apply a generic rumor detection model to report a decision of classification (e.g., rumor or non-rumor) at each of the predetermined checkpoints while leaving the determination of the best detection point to human judge. This is subject to a delayed decision as the results at the later checkpoints also need to be examined. 2) Methods that are trained to automatically determine an early detection point, but cannot guarantee the stability of decision (Zhou et al., 2019; Song et al., 2019). For example, CED (Song et al., 2019) decides an early detection point using a fixed probability threshold to assess if the current prediction is credible or not. However, prediction probability does not really reflect model's confidence (Guo et al., 2017), and such decision without properly modeling the uncertainty beyond the decision point may fail to give a timely and reliable detection because the prediction could flip over and over again afterwards with new posts flow in.
In this work, we propose a new method called Hawkes EArly Rumor Detection (HEARD) to model the stabilization process of rumor detection based on a Neural Hawkes Process (NHP) (Mei and Eisner, 2017), which can automatically determine when to make a timely and stable decision of detection. The basic idea is to construct a detection stability distribution over the expected future predictions based on a sequence of prior and current predictions, such that an optimal time point can be fixed without any delay for awaiting and checking the upcoming data beyond that point. Our main contributions can be summarized as follows<sup>1</sup>:
- We introduce BEARD, the first EARD-oriented dataset, collected by covering as much as possible the early-stage information relevant to the concerned claims.
- We propose HEARD, a novel EARD model
based on the NHP to automatically determine an optimal time point for the stable decision of early detection.
- Extensive experiments show that HEARD achieves more effective EARD performance as compared to strong baselines on BEARD and two commonly used general rumor detection datasets.
2.1 Early Rumor Detection
Despite extensive research on general rumor detection, early detection has not been studied well. Many studies claimed that their general detection models can be applied to early detection by simply fed with data observed up to a set of pre-determined checkpoints (Ma et al., 2016; Yu et al., 2017; Ma et al., 2017, 2018; Guo et al., 2018; Bian et al., 2020). Nevertheless, how to determine an optimal early detection point from many checkpoints is missing and non-trivial, as deciding when to stop often needs to check the data or model's outputs after the current checkpoint, causing delays of detection.
Some methods were claimed further as designed for early detection. Zhao et al. (2015b) proposed to gather related posts with skeptical phrases, and performed detection with cluster-based classifiers over real-time posts. Nguyen et al. (2017) developed a hybrid neural model for post-level representation and credit classification, which were incorporated with the temporal variations of handcrafted features for detecting rumors. Wu et al. (2017) clustered relevant posts and selected key features from clusters to train a topic-independent classifier for revealing emergent rumors. Xia et al. (2020) employed burst detection to segment an event into sub-events and trained an encoder for each sub-event representation for incremental prediction. None of the above methods really address the key issues of early detection as they lack mechanisms enforcing the earliness, and they cannot automatically fix an optimal detection point either.
ERD (Zhou et al., 2019) used deep reinforcement learning to enforce model to focus on early time intervals for the trade-off between accuracy and earliness of detection, and is the first EARD method that can automatically decide to stop or continue at a checkpoint. Song et al. (2019) proposed another EARD method called Credible Detection Point
$^{1}$ Dataset and source code are released at https://github.com/znhy1024/HEARD
(CED) using a fixed probability threshold to determine if detection process should stop depending on the credibility of current prediction. However, these models are unstable or of low confidence because the uncertainty of future predictions is not taken into account in training.
2.2 Rumor Detection Datasets
Quite a few rumor detection datasets based on social media posts relevant to a set of claims were released, such as TWITTER (Ma et al., 2016), PHEME (Zubiaga et al., 2016), RumourEval-2017/19 (Derczynski et al., 2017; Gorrell et al., 2019), FakeNewsNet (Shu et al., 2020), etc.. RumourEval-2017/19 are minor variants of PHEME while FakeNewsNet is never used for EARD. These datasets were built for general detection of rumors without much consideration on the earliness of information. Thus, the actual early-stage social engagements may not be covered by different data collection mechanisms used, such as applying a rigid time cut-off in Search API or launching a real-time gathering with Streaming API after a news outbreak. To our best knowledge, there is no dataset specifically built for early rumor detection task.
3 BEARD Corpus Construction
We scrape the text of title, claim, debunking time and veracity label in the articles on the fact-checking website snopes.com. Our goals are two-fold: 1) The collected posts are not only relevant to the claim but can diversely cover copious variations of relevant text expressions; 2) The collection can cover posts of early arrival, possibly ahead of the pertinent news exposure on the mainstream media.
To this end, we firstly construct high-quality search queries for Twitter search. An original query is formed from the title and claim of each article, with stop words removed. Since the lengthy query might harm the diversity of search results, we utilize some heuristics to obtain a substantial set of variants of each query potentially with better result coverage in Twitter search: i) We preform synonym replacement to create a set of variants of the query; ii) We shorten each variant by removing its words one by one with carefully crafted rules to maintain useful information, e.g., named entities, for good search quality, while keeping the remaining words after each removal as a new variant. As a result, we
obtain a substantial set of variants of the original query and merge the Twitter search results of each query and all its variants.
To cover early posts, each Twitter search is performed in an iterative fashion. To avoid groundtruth leakage, we first obtain the possible earliest official debunking time of the given claim by cross-checking its similar claims in a range of fact-checking websites (see Appendix A.1). From the earliest debunking time, we search backward for the relevant posts within  $M$  days prior to debunking, and then push back further  $N$  days earlier than before in each iteration until the number of newly gathered posts in an iteration becomes less than  $1\%$  of the posts obtained from the previous iteration.
Finally, for each retrieved post, we use its conversation ID to find the root post of the conversation it is engaged in. We utilize Sentence-BERT (Reimers and Gurevych, 2019) to retain those root posts with cosine similarities to the claim being higher than an empirical threshold. Thus far, we have obtained a set of conversation IDs for each claim which are led by different root posts (see Appendix A.4 for post-processing). Then we fetch from Twitter all the posts in the detected conversations along with the root posts into our final collection as an instance, and label the conversation as rumor if the corresponding claim is from the "Fact Checks" category on the Snopes or non-rumor if it is from the "News" category.
Due to space limit, we provide the details of search queries construction in Appendix A.2, and the settings of iterative Twitter search in Appendix A.3.
4 Problem Definition
Let  $\mathbf{C} = \{C\}$  denote a set of instances, where each  $C = \{y,S\}$  consists of the ground-truth label  $y\in \{0,1\}$  and a set of relevant posts in chronological order  $S = \left\{\left(\mathbf{m}_1,\tau_1\right),\dots ,\left(\mathbf{m}_i,\tau_i\right),\dots ,\left(\mathbf{m}_{|S|},\tau_{|S|}\right)\right.$ .  $y$  indicates  $C$  is a rumor if  $y = 1$  or a non-rumor otherwise.  $|S|$  is the number of relevant posts in  $S$ . Each tuple  $(\mathbf{m}_i,\tau_i)\in S$  includes the text content  $\mathbf{m}_i$  and the timestamp  $\tau_{i}$  of the  $i$ -th post, where  $\tau_{i}$  is defined as the time difference between the first and the  $i$ -th post, such that  $\tau_{1} = 0$  and  $\tau_{i - 1}\leq \tau_{i}$  for  $i > 1$ . In other words,  $\tau_{i}$  can be regarded as the elapsed time relative to the earliest post so that the timelines of different instances are aligned.
Figure 1: The architecture of HEARD. When the current observation arrives at time  $t_i$ , rumor detection predictions from LSTM  $\hat{y}_i$  and  $\hat{y}_{i-1}$  are used to update PI count  $I(t_i)$ . Then the intensity function  $\lambda(t)$  is computed by CTLSTM for  $t > t_i$ . HEARD will determine  $t_i$  as the earliest time point with stable prediction  $\hat{y}_i$  and stop if the stability prediction  $\hat{I}(t_{\infty})$  equals to  $I(t_i)$ , or continue otherwise.
Following the pre-processing method in most prior studies (Ma et al., 2016; Song et al., 2019; Zhou et al., 2019), we divide each posts sequence into a sequence of intervals to avoid excessively long sequence. We chop a sequence  $S$  into intervals based on three strategies: 1) fixed posts number, 2) fixed time length and 3) variable length in each interval (Zhou et al., 2019). Hence  $S$  is converted to  $X = \{(\mathbf{x}_i,t_i)\}_{i = 1}^{|X|}$ , where  $|X|$  is the number of intervals,  $\mathbf{x}_i = \{\mathbf{m}_{i,1},\mathbf{m}_{i,2},\dots,\mathbf{m}_{i,|\mathbf{x}_i|}\}$  and  $t_i = \tau_{i,|\mathbf{x}_i|}$  which is the timestamp of the last post  $\mathbf{m}_{i,|\mathbf{x}_i|}$  in the  $i$ -th interval. Then, we merge the posts falling into the same interval as a single post.
We define the EARD task as automatically determining the earliest time  $\hat{t} \in \{t_i\}$ , such that the prediction  $\hat{y} \in \{0,1\}$  at  $\hat{t}$  for a given claim is accurate and remains unchanged afterwards with time goes by. It is worthwhile to mention that since  $t_i$  relates to the granularity of intervals, it might affect the precision of a decision point based on the formed intervals. In practice, however, we will try to make the intervals small for keeping such impact marginal.
5 HEARD Model
Figure 1 shows the architecture of HEARD, which contains two components: 1) the rumor detection component predicts rumor/non-rumor label at each time step/interval; 2) the stabilization component models the prediction stabilization process and determines when to stop at the earliest detection point.
We will describe them with detail in this section.
5.1 Rumor Detection Modeling
A standard LSTM cell (Hochreiter and Schmidhuber, 1997) followed by a fully-connected layer is utilized for rumor detection in each interval. For any  $(\mathbf{x}_i,t_i)\in X$ $\mathbf{x}_i$  can be turned into a vector  $\mathbf{e}_i$  by a text representation method, e.g., TF-IDF (Salton and Buckley, 1988), CNN (Kim, 2014), BERT (Devlin et al., 2019), etc.. Taking  $\mathbf{e}_i$  as input, the LSTM cell gets the hidden state  $\mathbf{h}_i = \mathrm{LSTM}(\mathbf{e}_i)$  and forwards it through the fullyconnected layer to perform prediction. The predicted class probability distribution of an instance at  $t_i$  is calculated as  $\mathbf{p}_i^r = \sigma (\mathbf{Wh}_i + \mathbf{b})$  and thus the predicted class is  $\hat{y}_i = \operatorname {argmax}(\mathbf{p}_i^r)$  , where  $\sigma (\cdot)$  W and b are sigmoid function, weight matrix and bias, respectively.
5.2 Stabilization Process Modeling
Prediction Inverse (PI). Our rumor detection component keeps observing the posts stream and outputs a prediction sequence  $\hat{y}_1, \hat{y}_2, \hat{y}_3, \ldots$  along the time steps. During the process, newly arrived posts may provide updated features rendering the next decision of rumor detection to invert from rumor to non-rumor or the other way round. Presumably, the predictions would get stabilized when sufficient clues are accumulated over time. By modeling such a process, we aim to fix the earliest time  $t_i$  when the model can produce a stable prediction, meaning that there will be no expected inverses of prediction
occurring from  $t_i$  onward. Thus, we need to train the model for learning the expected future PIs to maximize its stability of prediction.
As a future PI can occur unforeseeably at any time, we introduce  $I(t)$  as a cumulative count of PIs up to any time point  $t$  in a continuous time space to accommodate the uncertainty. Then a PI counts sequence  $I(t_{1}), I(t_{2}), I(t_{3}) \ldots$  can be obtained from the prediction sequence. Clearly, we have  $I(t_{1}) = 0$ , and  $I(t_{i}) = I(t_{i-1}) + 1$  if  $\hat{y}_{i} \neq \hat{y}_{i-1}$  or  $I(t_{i}) = I(t_{i-1})$  otherwise. Additionally, we denote  $\mathcal{H}_{t_{i}} = \{(I(t_{1}), t_{1}), \dots, (I(t_{i-1}), t_{i-1})\}$  the history of PI counts up to  $t \in (t_{i-1}, t_{i}]$ , and denote  $\Delta I(t_{i}, t_{j}) = I(t_{j}) - I(t_{i})$  the difference of PI counts between  $t_{i}$  and  $t_{j}$  for  $j > i$ .
Neural Hawkes Process (NHP). General Hawkes process (Hawkes, 1971) is a doubly stochastic point process for modeling sequence of discrete events in continuous time, which has been successfully applied in social media research, such as modeling the popularity of tweets (Zhao et al., 2015a), rumor stance classification (Lukasik et al., 2016), fake retweeter detection (Dutta et al., 2020), and extracting temporal features for fake news detection (Murayama et al., 2020). Given a sequence of events  $E_{1}, E_{2}, E_{3}, \ldots$  which occur at the corresponding time points  $t_{1}, t_{2}, t_{3}, \ldots$  and  $N(t)$  that is the number of events occurring up to time  $t$ , a univariate Hawkes process with a conditional intensity that indicates expected arrival rate of future events at  $t$  is defined as (Yang and Zha, 2013):
$$
\lambda (t) = \mu + \int_ {- \infty} ^ {t} \kappa (t - s) \mathrm {d} N (s) \tag {1}
$$
where  $\mu \geq 0$  is the base intensity of event and  $\kappa (\cdot)$  is a manually specified monotonic kernel function that shows how the excitation from history decays with time. It assumes that arrived events can temporarily raise the probability of future events but the influence monotonically decays over time. However, this assumption is very strong which limits its ability for modeling complex dynamic point processes. In rumor diffusion, prediction inverse is the event influenced by many factors, such as what users express in historical and upcoming posts, which may bring tremendous uncertainty of prediction invalidating the monotonic decay assumption. Thus, we propose to adopt an NHP (Mei and Eisner, 2017) to capture the complex effects by utilizing a RNN with continuous-time LSTM (CTL
STM) to learn the intensity function. CTLSTM extends the vanilla LSTM with an interpolation-like mechanism so that its hidden state for controlling intensity can be updated discontinuously with each event occurrence and also evolves continuously as time elapses towards the next upcoming event.
Intensity Function Estimation. We use NHP to model the dynamics of PI (i.e., event) and approximate a detection stability distribution over the expected future predictions based on the sequence of historical and current predictions. As shown in Figure 1, CTLSTM reads the current observation  $(I(t_{i}),t_{i})$  to obtain  $\lambda (t)$  for any  $t > t_i$ , so that the distribution over the expected PI counts can be approximated. The intensity function  $\lambda (t)$  is controlled by a hidden state  $\widetilde{\mathbf{h}} (t)$  as follow:
$$
\lambda (t) = f \left(\widetilde {\mathbf {W}} \cdot \widetilde {\mathbf {h}} (t)\right) \tag {2}
$$
where  $\widetilde{\mathbf{W}}$  is a weight matrix and  $f(x) = \beta \log (1 + \exp (x / \beta))$  is the softplus function to obtain a positive intensity with a scale parameter  $\beta$  (Mei and Eisner, 2017).
To model the unknown future for  $t > t_i$  based on historical representation  $\widetilde{\mathbf{h}}(t_i)$ , a new hidden cell vector  $\widetilde{\mathbf{c}}(t)$  is introduced to control how  $\widetilde{\mathbf{h}}(t)$  continuously evolves over time. Specifically,  $I(t_i)$  is firstly transformed to a vector at  $t_i$  by a fully-connected layer to join the updates in CTLSTM. Then, CTLSTM updates the hidden state  $\widetilde{\mathbf{h}}(t)$  that has been evolving towards  $\widetilde{\mathbf{h}}(t_i)$  based on  $\widetilde{\mathbf{c}}(t)$  for  $t \in (t_{i-1}, t_i]$ . Thus, a richer representation of history can be learned by taking into account the dynamics of impact between the two consecutive observations. Meanwhile, to model the expected future PI count,  $\widetilde{\mathbf{c}}(t)$  is updated to a new state with the current cell input, which is analogous to how vanilla LSTM updates hidden cell², and from the new state,  $\widetilde{\mathbf{c}}(t)$  begins to continually approximate a target state that is defined by CTLSTM to represent an expected state of  $\widetilde{\mathbf{c}}(t)$  for  $t \to \infty$ .
Expected Stabilization.  $\lambda (t)$  indicates the expected instantaneous rate of future PIs from  $t$  onward. We can predict the value of  $I(t_{\infty})$  ( $t_{\infty}$  denotes  $t\to \infty$ ) and further determine an expected earliest stabilized observation ( $I(t_{i^*}),t_{i^*}$ ), such that  $t_{i^*} = \min \{t_i\mid \Delta I(t_i,t_\infty) = 0,i\in \{1,\ldots ,|X|\} \}$ . Hence,  $t_{i^*}$  indicates the expected
2The difference is that vanilla LSTM updates the hidden cell based on that of the previous time step while the update here is based on  $\widetilde{\mathbf{c}} (t)$  for  $t\to t_i$
earliest time that the predictions remain unchanged after it. We approximate  $\Delta I(t_i, t_\infty)$  by
$$
\Delta \hat {I} \left(t _ {i}, t _ {\infty}\right) = \int_ {t _ {i}} ^ {\infty} \lambda (t) \mathrm {d} t \tag {3}
$$
where  $\hat{I} (\cdot)$  constantly denotes a predicted PI count. We use Monte Carlo trick to handle all integral estimations (Mei and Eisner, 2017). Given  $t_{j*}$ , HEARD finally outputs  $\hat{y}_i^*$  as the stable prediction for early rumor detection.
5.3 HEARD Model Training
We utilize a next observation  $(I(t_{i + 1}),t_{i + 1})$  prediction task to train CTLSTM. With PIs as the indicator of model stability, CTLSTM aims to fit the sequence of observations  $\mathcal{H}_{t_{i + 1}}$  obtained from the predictions of the rumor detection module.
Specifically,  $\hat{I}(t_{i+1})$  can be obtained from  $I(t_i) + \Delta \hat{I}(t_i, t_{i+1})$ , and the difference value  $\Delta \hat{I}(t_i, t_{i+1})$  is either 1 or 0 which can be predicted as  $\Delta \hat{I}(t_i, t_{i+1}) = \operatorname{argmax}(\mathbf{p}_i^c)$ , where we infer  $\mathbf{p}_i^c = \sigma(\mathbf{W}'\widetilde{\mathbf{h}}(t_i) + \mathbf{b}')$  with trainable parameters  $\mathbf{W}'$  and  $\mathbf{b}'$ . For predicting  $t_{i+1}$ , a density  $p_i(t)$  is formulated as
$$
p _ {i} (t) = \lambda (t) \exp \left(- \int_ {t _ {i}} ^ {t} \lambda (s) \mathrm {d} s\right) \tag {4}
$$
Then we use the minimum Bayes risk predictor for time prediction (Mei and Eisner, 2017):
$$
\hat {t} _ {i + 1} = \mathbb {E} \left[ t _ {i + 1} \mid t _ {i}, \mathcal {H} _ {t _ {i}} \right] = \int_ {t _ {i}} ^ {\infty} p _ {i} (t) t \mathrm {d} t \tag {5}
$$