# Detecting Disinformation in Croatian Social Media Comments
## 4 Model Evaluation on Croatian Social Media Dataset
### Training the Classification Models
All three models exhibit a gradual increase in the true positive rate (TPR) across a broad range of false positive rate (FPR) values, indicating balanced trade-offs between sensitivity and specificity. However, none of the models demonstrate a steep initial rise in the ROC curve, highlighting their limited ability to achieve near-perfect class separation. The mid-range FPR values (approximately 0.3-0.6) present challenges for all models, as the TPR does not increase as sharply as expected, likely due to overlapping feature distributions. Additionally, the NN model's curve does not approach the upper-left corner sharply, suggesting it struggles to achieve high sensitivity at low false positive rates. While the \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Parameter** & **SVM** & **RF** & **NN** \\ \hline TPR & 0.6709 & 0.6962 & 0.7216 \\ FPR & 0.1176 & 0.1744 & 0.2235 \\ AUC & 0.78 & 0.76 & 0.75 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Parameters needed to plot AUC-ROC curves. Similar F1 scores for the three models--Support Vector Machine (SVM) with 74.62%, Random Forest (RF) with 73.84%, and neural network with 73.55%--suggest that the proposed framework performs consistently across different algorithms. This consistency indicates that the framework is robust and generalizes well across various machine learning techniques. However, the close F1 scores also highlight that the models may not sufficiently distinguish themselves in terms of performance. This could be due to the inherent limitations of the dataset, or the features used for training. To provide a further evaluation of model performance, a qualitative comparison of the models was made, and the results are presented in Table 3. While the AUC-ROC serves as a key measure of overall discriminatory ability, additional metrics such as accuracy, precision, recall, and F1 score offer deeper insights into each model's strengths and limitations. Among the three models, the SVM achieves the highest accuracy and precision, indicating its strong capability to correctly classify positive instances while minimizing false positives. This makes the SVM particularly well suited for scenarios where high precision is critical, such as medical diagnostics or fraud detection. However, its moderate recall suggests that it may struggle with correctly identifying all positive instances, potentially leading to higher false negatives in imbalanced datasets. The Random Forest model offers a balanced trade-off between precision and recall, with moderate-to-high performance across all metrics. While its AUC-ROC (0.76) is slightly lower than that of the SVM, it demonstrates robustness to noise and variability in feature importance, making it a more adaptable choice for real-world classification tasks. The model's ability to capture complex feature interactions contributes to its moderate-high recall, making it more effective in applications where missing positive cases must be minimized. However, its interpretability is lower compared to the SVM, and computational efficiency may decrease with large datasets. The neural network model exhibits the highest recall, making it particularly effective in scenarios where identifying all positive instances is a priority, even at the expense of increased false positives. This high recall performance is beneficial in applications such as anomaly detection and medical screening, where missing positive cases could have significant consequences. However, its moderate precision and accuracy indicate that it may produce a higher number of false positives, requiring careful postprocessing or threshold optimization. Furthermore, the NN model may require more extensive training data to generalize effectively, and its computational demands are typically higher than those of traditional machine learning models.