# Detecting Disinformation in Croatian Social Media Comments
## 4 Model Evaluation on Croatian Social Media Dataset
### Training the Classification Models
This result is slightly lower than the SVM model's performance but still demonstrates a reasonable level of effectiveness. The Random Forest's ability to handle many features and its robustness against overfitting are strengths, but the slightly lower accuracy suggests there may be room for optimization. To improve the Random Forest model's performance, we could consider increasing the number of trees in the forest, tuning hyperparameters such as the maximum depth of the trees or experimenting with different feature selection methods. Additionally, combining the Random Forest with other models in an ensemble approach might enhance overall accuracy. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Metric** & **SVM** & **RF** & **NN** \\ \hline Accuracy & 78.05\% & 76.36\% & 74.55\% \\ Precision & 84.13\% & 78.57\% & 75.00\% \\ Recall & 67.09\% & 69.62\% & 72.16\% \\ F1 score & 74.62\% & 73.84\% & 73.55\% \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Metric parameters for learning mechanisms. Figure 5: The confusion matrixes for (**a**) SVM, (**b**) Random Forest, and (**c**) neural network models. The neural network model is configured with one hidden layer consisting of 10 neurons. Each neuron in the hidden layer utilizes the ReLU (Rectified Linear Unit) activation function, which introduces non-linearity to the model and helps in learning complex patterns. To prevent overfitting, dropout regularization is applied with a dropout rate of 0.5, randomly setting 50% of the neurons to zero during each training iteration. This technique helps in improving the generalization of the model by reducing the reliance on specific neurons. Additionally, L2 regularization (also known as weight decay) is employed to further mitigate overfitting by penalizing large weights. The model is optimized using the stochastic gradient descent (SGD) algorithm, which iteratively updates the weights to minimize the loss function. This configuration demonstrates balanced classification performance, achieving a 75% accuracy and an F1 score of 73.55%. Its higher recall relative to precision suggests the model prioritizes detecting positive instances, which is crucial in applications like disease detection, fraud prevention, or security threat monitoring. However, the elevated false positive rate (190 instances) indicates that the model is more likely to generate false alarms, potentially leading to inefficiencies in practical deployment. While this accuracy is also slightly lower than that of the SVM and Random Forest models, it still demonstrates a reasonable level of effectiveness. The performance of the neural network could potentially be improved by experimenting with different network architectures, such as adding more hidden layers or neurons, or by using more advanced optimization algorithms like Adam or RMSprop. Additionally, fine-tuning the learning rate and other hyperparameters might enhance the model's accuracy. Further optimization and experimentation could help unlock its full potential. To further understand the models' discriminative capabilities, we include the Area Under the Receiver Operating Characteristic (AUC-ROC) curve as an additional and critical metric. To plot the AUC-ROC curves for each model--Support Vector Machine (SVM), Random Forest (RF), and neural network (NN)--specific parameters need to be calculated, including the true positive rate (TPR) and false positive rate (FPR). The parameters achieved in our study are presented in Table 2.  
Figures 6-8 show the AUC-ROC curves for the SVM, RF, and NN models, respectively. The performance of the three models--Support Vector Machine (SVM), Random Forest (RF), and neural network (NN)--as assessed by their AUC-ROC curves indicates moderate discriminatory power in classifying positive and negative instances. The SVM model achieves the highest AUC (0.78), suggesting slightly better class separation, followed by the Random Forest (0.76) and neural network (0.75) models.