# Collecting a Large Scale Dataset for Classifying Fake News Tweets Using Weak Supervision
## 3 Datasets
### Large-Scale Training Dataset
Since the Twitter API only returns the most recent 3200 tweets for each account ([[https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-user_timeline.html](https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-user_timeline.html)]([https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-user_timeline.html](https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-user_timeline.html)) (accessed on 26 April 2021), the majority of tweets in our dataset is from the year 2017, e.g., for an active twitter account with 20 tweets per day, that limitation Twitter API allows us retrieve tweets for the past 160 days (The new Twitter API v2, which is being rolled out since the beginning of 2021, has removed some of those limitations: [[https://developer.twitter.com/en/docs/twitter-api/early-access](https://developer.twitter.com/en/docs/twitter-api/early-access)]([https://developer.twitter.com/en/docs/twitter-api/early-access](https://developer.twitter.com/en/docs/twitter-api/early-access)) (accessed on 26 April 2021). In total, we collected 401,414 examples, out of which 110,787 (27.6%) are labeled as fake news (i.e., they come from fake news sources), while 290,627 (72.4%) are labeled as real news (i.e., they come from trustworthy sources). Figure 1 shows the distribution of tweets by their tweet time. Due to the collection time, the maximum tweet length is 140 characters, since the extension to 280 characters was introduced after our data collection ([[https://blog.twitter.com/en_us/topics/product/2017/tweetingmadeeasier.html](https://blog.twitter.com/en_us/topics/product/2017/tweetingmadeeasier.html)]([https://blog.twitter.com/en_us/topics/product/2017/tweetingmadeeasier.html](https://blog.twitter.com/en_us/topics/product/2017/tweetingmadeeasier.html)) (accessed on 26 April 2021). Figure 2 shows the topical distribution of the tweets. Figure 3 depicts further statistics about the tweets in the training set. It can be observed that while there is no strong difference in the sentiment (average 0.39 on the real class, 0.38 on the fake class) and the subjectivity score (average 0.27 on the real class, 0.29 on the fake class), the number of retweets (average 123 on the real class, 23 on the fake class) and favorites (average 236 on the real class, 34 on the fake class) differ considerably. It is important to point out that for collecting more than 400 k tweets, the actual annotation workload was only to manually identify \(111\) sources. In other words, from \(111\) human annotations (trustworthy vs. untrustworthy source), we produce 400 k annotated tweets. As discussed above, we expect the real news class to contain only a negligible amount of noise, but we inspected the fake news class more closely. The results are depicted in Table 1. The fake news class contains a number of actual news (a phenomenon also known as _mixed information_[24]), as well as tweets which are not news, but other contents (marked as \"no news\" in the table). These numbers show that the fake news tweets are actually the smallest class in the training sample. However, since the sample contains both real and fake news tweets from the same period of time, we can assume that for real news, those will also appear in the class labeled as non-fake, and since the real news class is larger by a factor of three, the classifier will more likely label them as real news. For example, if a real news item is tweeted by eight real and two fake news sources, a decent classifier would, to put it very simply, learn that it is a real news item with 80% confidence. This shows that the incidental imbalance of the dataset towards real news is actually useful. Figure 1: Distribution of tweets labeled as real and fake news in the training dataset. Figure 2: Topic distribution in the training dataset.