# Detecting Disinformation in Croatian Social Media Comments
## 4 Model Evaluation on Croatian Social Media Dataset
### Expanding the Framework: Testing with croBERT  
Our initial experiments with traditional machine learning techniques, such as Support Vector Machines (SVMs), Random Forests (RFs), and neural networks (NNs), achieved only modest F1 results. Another incentive was the need to mitigate the limitations posed by the domain specificity of the FRENK dataset. Those reasons prompted us to extend our research by integrating croBERT, a transformer-based model trained specifically for the Croatian language. CroBERT excels at capturing intricate linguistic structures, contextual relations, and semantic subtleties--capabilities that traditional machine learning models often lack, especially for tasks involving context-rich discourse in social media. Considering that disinformation often exploits subtle linguistic manipulations, the use of croBERT increases the sophistication of text analysis in this study. Furthermore, the inclusion of croBERT in our framework not only expands the scope of our experiments, but also provides a more robust power measure. Despite being fine-tuned on the same corpus, croBERT demonstrated a stronger ability to capture subtle, implicit, and context-dependent forms of disinformation compared to traditional models. Its performance on borderline and semantically complex instances suggests a degree of generalization beyond overt lexical cues, reinforcing the value of transformer-based architectures in handling complex and context-sensitive language, even within a domain-constrained setting. This brings our methodology in line with state-of-the-art natural language processing (NLP) techniques and underscores the scientific rigor and current relevance of our research. Finally, the comparative evaluation of traditional models and advanced transformers provides valuable insights into the trade-offs between model complexity, computational efficiency, and prediction accuracy, further enriching the results of this study. The croBERT experiments were executed on a MacBook Pro equipped with a 3.49 GHz 8-Core Apple M2 processor, 8 GB of RAM, and a 10-Core Apple GPU with shared memory. The fine-tuning process of an already pre-trained model required approximately 1 h and 20 min, which reflects the computational complexity inherent to transformer-based models in processing linguistic structures and contextual semantics. Conversely, the neural network (NN) model, trained under identical hardware conditions, completed its training within roughly 10 min. This discrepancy in training duration underscores the substantial computational demands associated with advanced transformer architectures relative to traditional neural networks, emphasizing the inherent trade-offs between model sophistication, resource intensity, and processing efficiency. The results (presented in Table 4) demonstrated a substantial improvement in performance compared to traditional machine learning models. While our previous experiments achieved a modest F1 score of around 0.74, croBERT yielded an F1 score of 0.89 after three epochs of training. We opted for a limited number of epochs due to practical constraints related to computational resources and time limitations. Fine-tuning transformer models can be computationally intensive, often requiring significant processing power and time. Given the resources available for this study, we focused on balancing thorough model evaluation with the efficient use of time and hardware. Despite the limited number of epochs, the model demonstrated strong and stable performance, with a clear downward trend in both training and evaluation loss. This approach ensured that we could still provide meaningful insights into croBERT's capabilities within the scope of our research. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Model** & **Accuracy** & **Precision** & **Recall** & **F1 Score** \\ \hline croBERT & 89.56\% & 90.31\% & 90.67\% & 89.33\% \\ \hline \hline \end{tabular}
\end{table} 
Table 4: CroBERT experiment results.