# Collecting a Large Scale Dataset for Classifying Fake News Tweets Using Weak Supervision
## 5 Evaluation
### Setting 2: Validation Against Gold Standard  
As discussed above, the more important setting validates the approach using a manually annotated gold standard. Since that gold standard dataset was collected independently from the training dataset, and is never used for training, feature selection, or parameter optimization, we can safely state that our approach is not overfit to that dataset.  
For the feature sets, feature selection methods, and parameter settings, we used the setups that worked best in the cross validation settings. More precisely, the feature sets and feature selection methods are those depicted in Tables 3 and 4.  
The results of that evaluation are shown in Tables 5 and 6. In contrast to the results in cross validation, the neural network learner performs best in that scenario, and voting across all learners does not improve the results.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Learner** & **\#F** & **FS** & **Prec.** & **Rec.** & **F1** \\ \hline NB & 614 & V & 0.8704 & 0.8851 & 0.8777 \\ \hline DT & 34 & G & 0.8814 & 0.9334 & 0.9067 \\ \hline NN & 372 & V & 0.8752 & 0.8861 & 0.8806 \\ \hline SVM & 595 & MI & **0.9315** & 0.7727 & 0.8447 \\ \hline XGB & 8 & G & 0.8946 & **0.9589** & **0.9256** \\ \hline RF & 790 & G & 0.8923 & 0.8291 & 0.8596 \\ \hline V & – & – & 0.9305 & 0.9416 & **0.9360** \\ \hline WV & – & – & 0.9274 & 0.9384 & 0.9329 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Cross validation results with tweet and user features. The best results are marked in bold.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Learner** & **\#F** & **FS** & **Prec.** & **Rec.** & **F1** \\ \hline NB & 409 & V & 0.7024 & 0.6080 & 0.6518 \\ \hline DT & 845 & – & 0.6918 & 0.6401 & 0.6649 \\ \hline NN & 463 & MI & 0.8428 & 0.6949 & 0.7618 \\ \hline SVM & 565 & MI & 0.8492 & 0.6641 & 0.7453 \\ \hline XGB & 106 & G & **0.8672** & **0.7018** & **0.7758** \\ \hline RF & 873 & G & 0.8382 & 0.6494 & 0.7318 \\ \hline V & – & – & 0.8650 & 0.7034 & 0.7759 \\ \hline WV & – & – & 0.8682 & 0.7039 & **0.7775** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Cross validation results with tweet features only, and number of features used (#F). Abbreviations: Learners—Naive Bayes (NB), Decision Tree (DT), Neural Network (NN), Support Vector Machine (SVM), XGBoost (XGB), Random Forest (RF); Feature Selection (FS), Voting (V), Weighted Voting (WV)—Variance Threshold (V), Recursive Elimination with Mutual Information (MI) and with Gini Index (G). The best results are marked in bold.  
Table 7 summarizes the best achieved results for each of the four settings. It is remarkable that the results are not much worse than those achieved in cross validation: with tweet features only, the best F1 score achieved is 0.77 (compared to 0.78), with both tweet and user features, the best F1 score is 0.90 (compared to 0.94). This proves that our approach of using a large-scale, noisy training dataset is actually valid and yields results of remarkable quality, even though the classifier employed was originally trained on a different task.