SlowGuess/NAACL_2022_OCR huggingface dataset:
where  $\mathbb{E}[\cdot ]$  is an estimator for choosing an optimal time point  $t$  to minimize the expectation of risks.
The overall loss consists of three terms on rumor detection, expected earliest stable time and CTL-STM. Concretely, given an instance  $C$  with input sequence  $X = \{(\mathbf{x}_i,t_i)\}$ , let  $\mathbf{y}$  be the one-hot encoding of ground-truth label (i.e., rumor or not). At each time of observation  $t_i$ , the cross-entropy loss between prediction and ground truth is defined as  $L_{i}^{r} = -\mathbf{y}\cdot \log \mathbf{p}_{i}^{r}$ . For the expected earliest stable time, the loss at  $t_i$  is defined as
$$
L _ {i} ^ {e} = \left| \Delta \hat {I} - \Delta I \right| - \log \left(1 - \frac {i}{| X |}\right) \tag {6}
$$
where the first term is the loss of  $\Delta I(t_i, t_\infty)^3$  approximation in Eq. 3, and the second term encourages the model to select an earliest time possible.
Dataset	Instances #	Posts #	AvgLen (hrs)
TWITTER	R	498	182,499	2,538
N	494	466,480	1,456
PHEME	R	1,972	31,230	10
N	3,830	71,210	19
BEARD	R	531	2,644,807	1,432
N	667	657,925	1,683
Table 2: Statistics of datasets. R: Rumor; N: Non-rumor.
The loss incurred from CTLSTM is given as
$$
L _ {i} ^ {c} = - \Delta \mathbf {I} \cdot \log \mathbf {p} _ {i} ^ {c} + | \hat {t} _ {i + 1} - t _ {i + 1} | \tag {7}
$$
where  $\Delta \mathbf{I}$  is the one-hot encoding of target  $\Delta I(t_i, t_{i+1})$ , the first term is the loss of next prediction inverse and the second term is the loss of time prediction for next observation.
Our objective is to minimize the cumulative loss up to  $t_{i^*}$  when the early detection decision is made:  $L = \frac{1}{i^*}\sum_{i=1}^{i^*}L_i$ , where  $L_i = L_i^r + L_i^e + L_i^c$ . We use stochastic gradient descent (SGD) mini-batch training over all training instances.
Datasets. We use BEARD, TWITTER (Ma et al., 2016) and PHEME (Zubiaga et al., 2016) datasets in the evaluation. BEARD contains 1,198 rumors and non-rumors reported during 2015/03-2021/01 with around 3.3 million relevant posts. We hold out  $20\%$  of instances for tuning, and the rest are randomly split with a ratio of 3:1 for training/test. Results are averaged over 5 splits. In Table 2, we show the statistics of TWITTER, PHEME and BEARD datasets.
Baselines. We compare HEARD with four state-ofthe-art baselines using their original source codes: 1) BERT (Devlin et al., 2019) is fine-tuned on the "earliest rumor detection" task (Miao et al., 2021), in which the early detection strategy is to output a prediction using only the first post of each instance. 2) CED (Song et al., 2019) uses a fixed probability threshold to check if the prediction is credible for determining the early detection point. 3) ERD (Zhou et al., 2019) uses a Deep Q-Network (DQN) to enforce the model to focus on early posts for determining the time point to stop and output the detection result. 4) STN (Xia et al., 2020) use a time-evolving network to represent
3For simplicity,  $\Delta I(t_i,t_{\infty})$  and  $\Delta \hat{I} (t_i,t_{\infty})$  are denoted as  $\Delta I$  and  $\Delta \hat{I}$  in Eq. 6, respectively.
state-independent sub-events in posts sequence for classifying claims at each checkpoint.
To balance the sequence length and granularity of time intervals, we pre-process posts sequences in the three datasets differently. We merge every 10 posts in BEARD and every 2 posts in TWITTER while we only merge the posts with the same timestamp in PHEME due to its generally short sequences. For each interval, both ERD and STN use pre-trained word embeddings to initialize the embedding matrix and fine-tune it in the training, while CED uses the TF-IDF method (Salton and Buckley, 1988), all of which follow the settings in the original papers (Song et al., 2019; Zhou et al., 2019; Xia et al., 2020). We also follow the original CED setting by using TF-IDF with 1,000 dimensions for representing the posts in each time interval.
The hidden size of standard LSTM is set to 128 with the dropout rate of 0.1, and the size of CTLSTM is set to 64. We pad all the sequences in a batch to the same length as the longest one, with the batch size of 16. We use the Adam (Kingma and Ba, 2015) with a learning rate of  $2e - 4$  for optimization. To avoid overfitting, we add a L2 regularization with the weight of  $1e - 4$ . All values are fixed based on the validation set.
Our model HEARD is implemented using Pytorch<sup>4</sup>. We use the original source codes of all the baselines: CED<sup>5</sup> and ERD<sup>6</sup> are implemented with TensorFlow; BERT<sup>7</sup> are implemented with Pytorch, and we use the base uncased pre-trained model; The code of STN is obtained directly from the authors of the original paper (Xia et al., 2020) which is implemented with Pytorch. All the experiments are conducted on a server with 4*12GB NVIDIA GeForce RTX 2080 Ti GPUs.
We use the general classification evaluation metrics accuracy and F1-score together with several EARD-specific metrics (see below) for evaluation.
Early Rate (ER) (Song et al., 2019) is defined as the utilization ratio of posts:  $ER = \frac{1}{|\mathbf{C}|}\sum_{C\in \mathbf{C}}\frac{i_C}{|C|}$
$^{4}$ https://pytorch.org/  
 $^{5}$ https://github.com/thunlp/CED  
 $^{6}$ https://github.com/DeepBrainAI/ERD  
 $^{7}$ https://github.com/huggingface/  
transformers
where  $\mathbf{C}$  is the test set,  $i_{C}$  implies the early detection decision is made at the  $i$ -th post in instance  $C$  and  $|C|$  is the number of posts in it. Lower ER means the model can detect rumors earlier.
Early Detection Accuracy Over Time (EDAOT). The metric of detection accuracy over time widely used (Ma et al., 2016; Zhou et al., 2019; Xia et al., 2020) is unsuitable for EARD models as it enforces a model to output a decision at each checkpoint whereas an EARD model can decide its own optimal decision point which may be earlier and more accurate than its output at the checkpoint. Our variant requires a model output result only when it cannot make an early decision before a given checkpoint while both accuracy and average time of decisions will be presented. Specifically, given a set of checkpoints at time  $\{t_1,\dots,t_k\}$ , at the  $j$ -th checkpoint, the detection accuracy is  $Acc_{t_j} = \sum_{C\in \mathbf{C}}\frac{\mathbb{1}_{\hat{y}_i^* = y}}{|C|}$ , where the binary function  $\mathbb{1}$  takes 1 if  $\hat{y}_{i^{*}} = y$  or 0 otherwise. And the average time of decisions is  $AvgT_{t_j} = \sum_{C\in \mathbf{C}}\frac{t_i^*}{|\mathbf{C}|}$ , where  $t_i^* \leq t_j$ , and  $t_i^* = t_j$  if the model cannot make a decision before  $t_j$ .
Stabilized Early Accuracy (SEA) is a newly defined comprehensive metric considering accuracy,earliness and stabilization:
$$
\begin{array}{l} S E A = \frac {1}{3 | \mathbf {C} |} \sum_ {C \in \mathbf {C}} \left[ \mathbb {1} _ {\hat {y} _ {i ^ {*}} = y} + \left(1 - \frac {i _ {C}}{| C |}\right) \right. \\ \left. + \left(1 - \frac {\Delta I \left(t _ {i ^ {*}} , t _ {| X |}\right)}{| X | - i ^ {*}}\right) \right] \\ \end{array}
$$
where the first term is the ratio of correctly predicted instances at the predicted time point  $t_{i^*}$  indicating accuracy, the second term is the ratio of posts after  $t_{i^*}$  indicating earliness, and the third term is the ratio of unchanged predictions after  $t_{i^*}$  indicating stability. The value of SEA is bounded in [0, 1] and higher SEA means better performance.
The main results are provided in Table 3. We show the EDAOT results in Figure 2.
Results of Classification. STN uses the entire timeline of each instance since it cannot automatically determine an early detection point. As shown in Table 3, however, it only achieves comparable Accuracy and F1 as ERD and CED and is much worse than HEARD, even though it was reported much better than CED on TWITTER in previous
(a) TWITTER
(b) PHEME
Figure 2: Early detection accuracy over time within 48 hours. X-axis: timeline; Y-axis: accuracy; Vertical lines: checkpoints; Dots with shapes: decision points; Time above a dot: Deadline (i.e., checkpoint) set for the model.
(c) BEARD
Dataset	Model	Acc	F1	ER	SEA
TWITT	BERT	0.623	0.599	0.026	0.768
ERD	0.696	0.699	0.999	0.566
STN	0.682	0.649	1.000	0.561
CED	0.685	0.682	0.811	0.620
HEARD	0.716	0.714	0.348	0.789
PHEME	BERT	0.839	0.820	0.163	0.830
ERD	0.784	0.753	0.976	0.602
STN	0.810	0.787	1.000	0.603
CED	0.800	0.695	0.884	0.638
HEARD	0.823	0.805	0.284	0.841
BEARD	BERT	0.565	0.452	0.091	0.758
ERD	0.709	0.708	1.000	0.570
STN	0.711	0.690	1.000	0.570
CED	0.769	0.740	0.674	0.689
HEARD	0.789	0.788	0.490	0.765
Table 3: Early rumor detection results.
work (Xia et al., 2020) $^{8}$ . This implies that the early detection models are promising which can use a prior fraction of posts to achieve similar or much better results. It also suggests that capturing early-stage features is important to more accurate rumor detection. Only using the source post for detection, BERT gives worst Accuracy and F1 on TWITTER and BEARD, but it gets unexpectedly high performance on PHEME. We inspect this issue by following the prior analysis (Schuster et al., 2019) based on Local Mutual Information (LMI) (Evert, 2005) and Pointwise Mutual Information (PMI) (Church and Hanks, 1990). We find that the source posts in PHEME have spuriously much stronger correlation with the class labels. Appendix B.1 discusses such bias and the possible cause. This observation suggests data sampling bias exists in the existing dataset, and thus the model's decision based on the source post only can be misleading and insufficient.
Results of ER and SEA. Table 3 also shows that HEARD consistently outperforms ERD and CED in large margin based on ER and SEA, indicating HEARD is more effective and stable. HEARD considerably improves CED by  $57\%$ ,  $68\%$  and  $27\%$  in ER and by  $27\%$ ,  $32\%$  and  $11\%$  in SEA on TWITTER, PHEME and BEARD, respectively. ERD's high ER scores entails that it can hardly make early decision, as this DQN-based model seems weak on its reward function, which gives a small penalty to continuation but a large one to termination with wrong predictions (Zhou et al., 2019), discouraging it from stopping early. BERT only uses the first post for detection which thus obtains the lowest ER. Note that a model being expectantly stable at the time of decision means the prediction will remain unchanged even though new data could be seen by the model after that. To probe its stability, we enforce model to continue outputting predictions at the checkpoints later than its decision point. We can see that HERAD still outperforms BERT on SEA on all the datasets indicating it is more stable.
Results of EDAOT. Figure 2 shows that HEARD is clearly superior over the baselines in accuracy, earliness and stability at the checkpoints. ERD looks relatively stable but it hardly makes early detection due to aforementioned reason. Our conjecture is that its DQN module forces it to overly focus on the first few intervals while deferring the decision to the end due to the weak reward design, resulting in nearly no improvement even with more data. Note that in EDAOT evaluation a model should stop once it outputs a prediction for a given checkpoint (i.e., deadline). Thus, BERT reports all the predictions using only the first post, rendering the same accuracy at checkpoints. Interestingly, HEARD can make especially fast decisions on PHEME which only uses a little less than 6 hours given a 48-hour deadline. The reason might be the average length
The CED performance reported in (Xia et al., 2020) is an excerpt from (Song et al., 2019) based on half of the TWITTER data, while they experimented STN using the full data.
(a) Germanwings
(b) Darkest baby
Figure 3: (a) A rumor in PHEME on "The co-pilot of Germanwings flight 9525 was Muslim convert"; (b) A rumor in TWITTER on "A photograph shows the darkest baby in the world". Y-axis: the number of collected posts; X-axis: timeline in minutes (m)/days (d); Colored bar: the semantic similarity between the posts in any time step and the posts in the decision time step, where the red bar denotes recollected data and the blue one denotes original data. The similarity ranges from 0.7 to 1.0, and the darker the color the more similar the posts are.
of instances in PHEME is only around 14 hours as compared to over 1,000 hours in other two datasets, which HEARD can especially benefit from in the training. However, considering the fact that the PHEME (and TWITTER) dataset may fail to cover the real early information, such a very quick detection on it could be an illusion since the patterns the model actually uses for the decision are from the midst of rumor propagation.
In this section, we give an intuitive illustration to reveal 1) why BEARD is more suitable for EARD task; and 2) how HEARD is more advantageous than the state-of-the-arts with automatic early time point determination. Specifically, we recollect the relevant posts of rumor cases in PHEME and TWITTER using our data construction method for BEARD. To analyze the first question, we inspect the posts before and after the recollection as shown in Figure 3. In the original posts, the actual early-stage timelines of both cases are largely missing. Our recollected data can cover the important denial and questioning posts signaling rumors in the early stage. This observation indicates that our data collection method has improved coverage by recalling these actual early information which is not available in the existing datasets.
To analyze the second question, we display the detection outputs of different models before and after the recollection. HEARD consistently detects rumors with less time than CED and ERD since it automatically terminates when stabilization process estimates the expected future prediction inverse
will not change. The correctness of such expectation could be reflected by the value of semantic similarity, which almost has no change with new posts coming in after the detection point, implying the future prediction results are expected stable.
8 Conclusion and Limitation
We introduce BEARD, a new benchmark dataset collected for early rumor detection, and propose a model called HEARD to perform stable early rumor detection based on neural Hawkes process. Experiments show that HEARD achieves overall better performance than state-of-the-art baselines. Analysis entails BEARD is more suitable for the early rumor detection task.
There are some cases that a rumor seems to be stable over an extended time period but is eventually refuted by an authoritative source later. It is because sensational discussions are more attractive to some social media users than facts, which leads to signals in support of the rumor are usually much stronger in social media, and the final correction often has only small engagement. In such cases, our model HEARD based on social media alone could be misled by the signals of only one source. In future, we plan to study incorporating authoritative sources of information to alleviate this phenomenon on social media, such as statements of official platforms, scientific sources, etc..
This research is supported by the National Research Foundation, Singapore under its Strategic Capabilities Research Centres Funding Initiative and the
Singapore Ministry of Education (MOE) Academic Research Fund (AcRF) Tier 1 grant. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of funding agencies.
Tian Bian, Xi Xiao, Tingyang Xu, Peilin Zhao, Wenbing Huang, Yu Rong, and Junzhou Huang. 2020. Rumor detection on social media with bi-directional graph convolutional networks. In Proceedings of the 34th AAAI conference on artificial intelligence, pages 549-556.
Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. O'Reilly Media, Inc.
Kenneth Ward Church and Patrick Hanks. 1990. Word association norms, mutual information, and lexicography. Computational Linguistics, 16(1):22-29.
Leon Derczynski, Kalina Bontcheva, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Arkaitz Zubiaga. 2017. SemEval-2017 task 8: RumourEval: Determining rumour veracity and support for rumours. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 69-76, Vancouver, Canada. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Hridoy Sankar Dutta, Vishal Raj Dutta, Aditya Adhikary, and Tanmoy Chakraborty. 2020. Hawkeseye: Detecting fake retweeters using hawkes process and topic modeling. IEEE Transactions on Information Forensics and Security, 15:2667-2678.
Stefan Evert. 2005. The statistics of word cooccurrences: word pairs and collocations. Ph.D. thesis, University of Stuttgart.
Genevieve Gorrell, Elena Kochkina, Maria Liakata, Ahmet Aker, Arkaitz Zubiaga, Kalina Bontcheva, and Leon Derczynski. 2019. SemEval-2019 task 7: RumourEval, determining rumour veracity and support for rumours. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 845–854, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 1321-1330.
Han Guo, Juan Cao, Yazi Zhang, Junbo Guo, and Jintao Li. 2018. Rumor detection with hierarchical social attention network. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pages 943-951.
Alan G Hawkes. 1971. Spectra of some self-exciting and mutually exciting point processes. Biometrika, 58(1):83-90.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746-1751, Doha, Qatar. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations.
Yang Liu and Yi-fang Brook Wu. 2018. Early detection of fake news on social media through propagation path classification with recurrent and convolutional networks. In Proceedings of the 32nd AAAI conference on artificial intelligence, pages 354-361.
Michal Lukasik, P. K. Srijith, Duy Vu, Kalina Bontcheva, Arkaitz Zubiaga, and Trevor Cohn. 2016. Hawkes processes for continuous time sequence classification: an application to rumour stance classification in Twitter. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 393-398, Berlin, Germany. Association for Computational Linguistics.
Jing Ma, Wei Gao, Prasenjit Mitra, Sejeong Kwon, Bernard J. Jansen, Kam-Fai Wong, and Meeyoung Cha. 2016. Detecting rumors from microblogs with recurrent neural networks. In Proceedings of the 25th International Joint Conference on Artificial Intelligence, pages 3818-3824.
Jing Ma, Wei Gao, and Kam-Fai Wong. 2017. Detect rumors in microblog posts using propagation structure via kernel learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 708-717, Vancouver, Canada. Association for Computational Linguistics.
Jing Ma, Wei Gao, and Kam-Fai Wong. 2018. Rumor detection on Twitter with tree-structured recursive neural networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1980–1989, Melbourne, Australia. Association for Computational Linguistics.
Hongyuan Mei and Jason Eisner. 2017. The neural hawkes process: A neurally self-modulating multivariate point process. In Advances in Neural Information Processing Systems 30, pages 6754-6764.
Xin Miao, Dongning Rao, and Zhihua Jiang. 2021. Syntax and sentiment enhanced bert for earliest rumor de
tection. In CCF International Conference on Natural Language Processing and Chinese Computing, pages 570-582.
Taichi Murayama, Shoko Wakamiya, and Eiji Aramaki. 2020. Fake news detection using temporal features extracted via point process. In Workshop Proceedings of the 14th International AAAI Conference on Web and Social Media.
Tu Ngoc Nguyen, Cheng Li, and Claudia Niederée. 2017. On early-stage debunking rumors on twitter: Leveraging the wisdom of weak learners. In International Conference on Social Informatics, pages 141-158.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Natali Ruchansky, Sungyong Seo, and Yan Liu. 2017. CSI: A hybrid deep model for fake news detection. In Proceedings of the 26th ACM International Conference on Information and Knowledge Management, pages 797-806.
Gerard Salton and Christopher Buckley. 1988. Termweighting approaches in automatic text retrieval. Information processing & management, 24(5):513-523.
Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. 2019. Towards debiasing fact verification models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3419-3425, Hong Kong, China. Association for Computational Linguistics.
Kai Shu, Deepak Mahudeswaran, Suhang Wang, Dongwon Lee, and Huan Liu. 2020. Fakenewsnet: A data repository with news content, social context, and spatiotemporal information for studying fake news on social media. *Big data*, 8(3):171–188.
Changhe Song, Cheng Yang, Huimin Chen, Cunchao Tu, Zhiyuan Liu, and Maosong Sun. 2019. Ced: Credible early detection of social media rumors. IEEE Transactions on Knowledge and Data Engineering, 33(8):3035-3047.
Liang Wu, Jundong Li, Xia Hu, and Huan Liu. 2017. Gleaning wisdom from the past: Early detection of emerging rumors in social media. In Proceedings of the 2017 SIAM International Conference on Data Mining, pages 99-107.
Rui Xia, Kaizhou Xuan, and Jianfei Yu. 2020. A state-independent and time-evolving network for early rumor detection in social media. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language
Processing (EMNLP), pages 9042-9051, Online. Association for Computational Linguistics.
Shuang-Hong Yang and Hongyuan Zha. 2013. Mixture of mutually exciting processes for viral diffusion. In Proceedings of the 30th International Conference on Machine Learning, volume 28, pages 1-9.
Feng Yu, Qiang Liu, Shu Wu, Liang Wang, and Tieniu Tan. 2017. A convolutional approach for misinformation identification. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 3901-3907.
Qingyuan Zhao, Murat A. Erdogdu, Hera Y. He, Anand Rajaraman, and Jure Leskovec. 2015a. SEISMIC: A self-exciting point process model for predicting tweet popularity. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1513-1522.
Zhe Zhao, Paul Resnick, and Qiaozhu Mei. 2015b. Enquiring minds: Early detection of rumors in social media from enquiry posts. In Proceedings of the 24th International Conference on World Wide Web, pages 1395-1405.