# Collecting a Large Scale Dataset for Classifying Fake News Tweets Using Weak Supervision
## 5 Evaluation
### Comparison to Training on Manually Labeled Data  
In order to be able to better judge the results above, we also ran a set of experiment where we use cross validation on the evaluation set, i.e., we use the manually labeled data as training signal. The goal of this experiment is to analyze how close the quality using weak supervision would get to results achieved through human expert labeling, which is expected to yield superior results. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Learner** & **Precision** & **Recall** & **F1-Measure** \\ \hline NB & 0.8333 & **0.9052** & 0.8678 \\ \hline DT & 0.8721 & 0.6466 & 0.7426 \\ \hline NN & **0.9115** & 0.8879 & **0.8996** \\ \hline SVM & 0.8942 & 0.8017 & 0.8455 \\ \hline XGB & 0.8679 & 0.7931 & 0.8288 \\ \hline RF & 0.8713 & 0.7586 & 0.8111 \\ \hline V & 0.8870 & 0.8793 & 0.8831 \\ \hline WV & 0.8761 & 0.8534 & 0.8646 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Results on manually labeled gold standard, using both tweet and user features. The best results are marked in bold. \begin{table}
\begin{tabular}{c c c} \hline \hline  & **Identifying Fake News...** \\ \hline  & **Sources** & **Tweets** \\ \hline Tweet features only & 0.7775 (Table 3) & 0.7699 (Table 5) \\ \hline Tweet and user features & 0.9360 (Table 4) & 0.8996 (Table 6) \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Results summary, depicting the best F1 score achieved for each task (fake news source and fake news tweet detection), as well as for each feature group (tweet level features only and tweet and source features). \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Learner** & **Precision** & **Recall** & **F1-Measure** \\ \hline NB & 0.6835 & 0.4655 & 0.5538 \\ \hline DT & 0.7895 & 0.6466 & 0.7109 \\ \hline NN & 0.7909 & **0.7500** & **0.7699** \\ \hline SVM & 0.7961 & 0.7069 & 0.7489 \\ \hline XGB & 0.8333 & 0.6897 & 0.7547 \\ \hline RF & **0.8721** & 0.6466 & 0.7426 \\ \hline V & 0.8200 & 0.7069 & 0.7593 \\ \hline WV & 0.8478 & 0.6724 & 0.7500 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Results on manually labeled gold standard, using tweet features only. The best results are marked in bold. Here, we distinguish two cases:  
1. In the first setup, we have re-created all content features for the manually labeled dataset (On the manually labeled dataset, LDA was not able to generate meaningful results due to the very small dataset size. Hence, we have left out the LDA based features in the first setup). 2. In the second setup, we used the distributional text and topic features (doc2 vec, LDA) created from the weakly supervised set. The latter case can be considered a _combined_ setup, where information from the large dataset obtained through weak supervision was used to improve the results on the manually labeled dataset. The parameter settings were the same as in the experiments on the large-scale training data. The results are depicted in Tables 8 and 9. They show that the F1 scores are in the same range when using the manually labeled tweets as when using the ones labeled through weak supervision--when using tweet features only, training on the manually labeled data works slightly better, when using all features, training on the weak supervision set works slightly better. This supports our claim that our methodology for generating training data is valid in the sense that it yields datasets of comparable quality in downstream classification tasks. In our second setup, we use the distributional features created on the large-scale dataset. Here, our intuition is that approaches such as LDA and doc2 vec yield good results only when trained on larger corpora. When using those features for classification with the manually labeled data, we obtain the best results overall, surpassing both the setup with manually labeled data only and the setup with labeling through weak supervision only by 3-4 percentage points in F1 score.