SlowGuess/NAACL_2022_OCR huggingface dataset:
Kaimin Zhou, Chang Shu, Binyang Li, and Joy Han Lau. 2019. Early rumour detection. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1614-1623, Minneapolis, Minnesota. Association for Computational Linguistics.
Arkaitz Zubiaga, Maria Liakata, and Rob Procter. 2016. Learning reporting dynamics during breaking news for rumour detection in social media. ArXiv preprint, abs/1610.07363.
A Corpus Construction Protocols
A.1 Claims Collection
Snopes<sup>9</sup> is a well-known fact-checking website where fact-checkers manually collect check-worthy claims from multiple sources (e.g., social media, e-mail, news, etc.) and review each claim to report a decision in terms of "Fact Checks" (i.e., rumors being fact-checked) or "News" (i.e., non-rumors of no need to check). Further, the checkers verify the rumors and compose detailed fact-checking articles for justifying the veracity of rumors.
We utilize Snopes to collect the claims for constructing our data instances in terms of rumors and non-rumors. There are around  $10\mathrm{k + }$  claims on Snopes in total (up to the end of 2021), but majority of them have very limited exposure on Twitter. Specifically, we include the claims into our collection based on the following principles: (1)
$9$ https://www.snopes.com/
We only include the claims that were published in recent 6 years since they have relatively more complete exposure on Twitter (e.g., aged posts might be more likely to get deleted); (2) We only include the claims that have relevant posts on Twitter before the claim was officially debunked. As a result, we collect 531 rumor and 667 non-rumor claims reported during 2015/03-2021/01. For each claim, we scrape the text of article title, the claim, the debunking time and the veracity label.
We also try to exclude posts that may leak the ground truth since a claim might have been fact-checked by other fact-checking websites, thus the truth might have been referenced in the social media posts, such as the rumor example in TWITTER shown in Table 1. To minimize the chance of ground truth leakage, therefore, we match each claim from Snopes across the claims from a handful set of fact-checking websites including FactCheck.org<sup>10</sup> and PolitiFact.com<sup>11</sup> to get the possible earliest official debunking time, from which we begin collecting the relevant posts backwards in time.
A.2 Query Construction
For each claim, we construct a set of high-quality search queries for Twitter search to diversely cover relevant posts. Firstly, we concatenate the title and claim of an article followed by stop words removal to filter out noise, resulting in an original query. Since the original query might be long and harm the diversity of search results, we carry out the following ad hoc operations to shorten the query for maintaining maximum useful information and possibly broadening the coverage of retrieved posts: (1) We perform synonym replacement with Natural Language Toolkit (NLTK) (Bird et al., 2009) to create a variant of the query for each replacement; (2) For each variant obtained in (1), we use Google Search API<sup>12</sup> to search for this altered query and rank by the frequencies of the highlighted words that are hit in the top-100 searched snippets; (3) We then shorten the original query and its variants obtained in (2) by removing the highlighted words that are contained in the queries one by one starting from the low-frequency words, while keeping the remaining words after each removal as a variant of
the query, until the shortest variant is left with three words. Note that here we perform named entity recognition by NLTK on the original query and the words that are parts of named entities will not be removed as they are useful for search.
A.3 Iterative Twitter Search
To prevent early quit of iteration caused by the intermittent sparse distribution of posts along the timeline, we manually adjust the values of  $M$  and  $N$  by trial and error with different instances to gather as much early posts as we can. We also manually check the search results based on a sample of instances using different settings of the termination threshold, i.e., the ratio of the number of gathered posts in each iteration over that in the previous iteration. We observe that too high threshold hinders early posts to be searched out, but too low threshold tends to introduce more noise. We finally set  $1\%$  as the termination threshold by trading off earliness and noise.
A.4 Posts Collection
Rather than merging the conversations led by different root posts regarding the same claim into a large conversation, it is more realistic to remain them naturally separated since the conversations originate from different sources and the merge may introduce unnecessary bias. And we drop all the posts that are published after the official debunking time of each claim to avoid ground truth leakage.
B.1 Bias Evaluation on Datasets
As mentioned in Section 6.4, we utilize LMI and PMI to examine the data sampling bias in the existing datasets and BEARD. The top-5 LMI-ranked words of PHEME, TWITTER and BEARD are shown in Table 4. The top-5 LMI-ranked words of PHEME have much higher LMI and PMI than the other two datasets indicating the high correlations between the words of source posts and the label. Meanwhile, the words in PHEME, e.g., 'breaking', 'hostages', 'shot', etc., are more eyes-catching comparing to the top ranked words in other two datasets. These idiosyncrasies might be introduced by the construction method of PHEME in a sense that journalists might see a timeline of posts about the breaking news and then annotate source posts of conversations, which are easily utilized by BERT to obtain high classification performance. Some
10https://www.factcheck.org/ 11https://www.politifact.com/ 12https://developers.google.com/ custom-search/v1/overview
Dataset	Word	LMI·10-6	PMI
PHEME	breaking	2,720	0.75
hostages	2,569	0.66
soldier	2,378	1.00
shot	2,308	0.79
cafe	2,283	0.74
TWITT	not	1,214	0.38
black	780	0.45
flag	745	0.49
trump	637	0.39
baby	589	0.53
BEARD	he	1,246	0.38
they	722	0.26
his	665	0.26
by	598	0.19
biden	580	0.51
Table 4: Top-5 LMI-ranked words of source posts correlated with class label in three datasets.
sallient bias also exists in TWITTER dataset evidenced as some top words with high LMI and PMI, such as 'black (lives)', 'flag' and 'trump', while we do not observe such bias among the top words in BEARD.