# Are Strong Baselines Enough? False News Detection with Machine Learning
## 3 Results
### Experiment 2: LIAR Dataset  
We used the LIAR dataset [4] for our second set of experiments. This dataset was already divided into training, validation, and testing files. We combined the validation and training data for these experiments, as the inclusion of the validation file was unnecessary for our experiments. To be able to use the ML classifiers, we first built a bag-of-words model with CountVectorizer and TfidfTransformer, like in the previous experiment. For the experiments, we used three different ML classification algorithms: _passive aggressive classifier_ (PA), _random forest_ (RF), and _support vector machine_ (SVM). As the LIAR dataset originally separates false and true news into six different labels, we concluded two different types of classification experiments in this dataset. At first, we concluded a six-label classification experiment, where we classified articles with the six different labels originally provided by the dataset's author. After that, we concluded a series of different binary classification experiments, where we reduced the labels of the dataset into only \"true\" and \"false\". We reduced the labels in five different ways: all labels except \"true\" are labeled as \"false\"; labels \"pants-fire\" and \"false\" are labeled as \"false\" and the rest as \"true\"; all labels except \"pants-fire\" are labeled as \"true\"; labels are split from the middle into \"true\" and \"false\"; and labels \"true\" and \"mostly-true\" are labeled as \"true\" and the rest as \"false\". We used these labeling methods to determine when non-binary classification works best for the LIAR dataset and which labels make the language the most differentiable. When performing the six-label classification, a split ratio of 90:10 was employed as that is how the dataset was split by the original author. During the multiple binary-label studies, we divided the dataset into a split ratio of 80:20. The results for the six-label classification experiment can be found in Table 12, and the ROC curve for this experiment can be observed in Figure 7. From these results, we can observe that all the used methods achieved low scores overall. The RF method achieved the best accuracy score; however, the FPR and FNR scores for this method were not good and showcased severe bias for detecting negative classes only. The PA and SVM  
Figure 6: ROC curve for the classifying results conducted using the Twitter15 dataset as test data and the LIAR as train data. methods performed more consistently, especially in terms of FPR and FNR scores, but their performance was not high either. After the six-way classification experiment, we concluded a series of binary classification experiments. The results for the binary classification experiments can be found in Tables 13-17. The ROC curves for these experiments can be found in Figures 8-12. From these tables, we can observe that the classifiers were most efficient when only the label \"pants-fire\" was considered as false and least efficient when the labels \"mostly-true\" and \"true\" were considered true and the rest were false. Overall, in the binary classification experiments, the classifiers were more accurate than in the six-label experiment. The methods in general performed equally, but in some binary classification experiments, the RF method again showed severe bias towards detecting only negative classes, whereas PA and SVM were more consistent with their FPR and FNR scores. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Method** & **Accuracy** & **Precision** & **Recall** & **F1** & **AUC** & **FPR** & **FNR** \\ \hline PA & 0.56 & 0.55 & 0.55 & 0.55 & 0.59 & 0.37 & 0.53 \\ RF & 0.59 & 0.59 & 0.59 & 0.59 & 0.63 & 0.45 & 0.35 \\ SVM & 0.58 & 0.58 & 0.58 & 0.58 & 0.60 & 0.25 & 0.62 \\ \hline \hline \end{tabular}
\end{table} 
Table 16: Performance report for the binary classification experiment where labels “barely-true”, “pants-fire” and “false” are considered false, and labels “half-true”, “mostly-true” and “true” are considered true.