# Are Strong Baselines Enough? False News Detection with Machine Learning
## 3 Results
### Experiment 1: Twitter15 Dataset  
For our first set of experiments, we used the Twitter15 dataset [36]. This dataset was already divided into separate \"test\" and \"train\" files. As this dataset was created for rumor detection, we used this dataset to compare the differences and similarities between false news and rumors. As presented in Table 1, the definitions of rumors and false news have a slight overlap; we thought that it would be interesting to conduct a classifying experiment using both rumors and false news. In this experiment, we first conducted a classifying experiment only using the Twitter15 dataset [36]. Then, we experimented using the LIAR dataset [4] as train data and the Twitter15 dataset [36] as test data. For the final experiment, we used the Twitter15 dataset [36] as train data and the LIAR dataset [4] as test data.  
We first built a bag-of-words model with _CountVectorizer_ and _Tfid/Transformer_ to use the ML classifiers. For this experiment, to obtain the best comparisons between all the different methods, we used all the methods introduced in Section 2.3.2. For the later experiments, we chose the three best-performing methods from this experiment.  
When only the Twitter15 dataset [36] was used in the tests, the split ratio was around 75:25, as specified by the original authors. Using the LIAR dataset [4] as either the \"train\" or \"test\" file, along with the Twitter15 dataset [36], the split ratio was 90:10.  
Our first experiment was first conducted only using the Twitter15 dataset [36] on the classifiers. This allowed us to create a decent baseline for our classifiers, which we could then compare to the classifying results obtained later by more complex studies. The results of this experiment can be seen in Table 9, and the ROC curve for this experiment can be observed in Figure 4. From Table 9, we can observe that all the classifiers performed well on average, as all the classifiers could achieve accuracy results of over 50%. The worst-performing classifiers were AdaBoost and MLP, which achieved accuracy scores of 54% and 68%. These classifiers did not perform well with AUC, FPR, and FNR scores either. Especially when looking at the FPR and FNR scores, we can observe that these classifiers had the most tendencies to misclassify the labels. The best-performing classifiers were the passive aggressive classifier and SVM, which achieved accuracy scores of 87%.  
The second and third experiments were conducted using the Twitter15 dataset as train data and the LIAR dataset as test data, and vice versa. The results for these experiments can be seen in Tables 10 and 11, and the ROC curves for these experiments can be observed in  
Figure 4: ROC curve for the classifying results conducted only using the Twitter15 dataset.  
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Method** & **Accuracy** & **Precision** & **Recall** & **F1** & **AUC** & **FPR** & **FNR** \\ \hline SGD & 0.84 & 0.84 & 0.84 & 0.84 & 0.93 & 0.04 & 0.15 \\ PA & 0.86 & 0.87 & 0.87 & 0.87 & 0.95 & 0.04 & 0.12 \\ RF & 0.81 & 0.82 & 0.80 & 0.80 & 0.94 & 0.06 & 0.43 \\ MLP & 0.68 & 0.70 & 0.68 & 0.68 & 0.50 & 0.33 & 0.66 \\ LR & 0.85 & 0.86 & 0.85 & 0.86 & 0.96 & 0.03 & 0.31 \\ ADA & 0.55 & 0.54 & 0.54 & 0.53 & 0.72 & 0.00 & 0.97 \\ kNN & 0.79 & 0.79 & 0.79 & 0.78 & 0.95 & 0.05 & 0.15 \\ NB & 0.80 & 0.80 & 0.80 & 0.80 & 0.96 & 0.06 & 0.36 \\ DT & 0.71 & 0.72 & 0.72 & 0.72 & 0.83 & 0.11 & 0.22 \\ SVM & 0.87 & 0.87 & 0.87 & 0.87 & 0.96 & 0.03 & 0.16 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Performance report on the classifying results conducted only using the Twitter15 dataset.  
Based on these classification experiments, we concluded that RF, PA, and SVM were the best-performing methods and the most consistent with their precision, recall, F1, AUC, FPR, and FNR scores. We use these three methods for the later classification experiments for these reasons.