# Detecting Disinformation in Croatian Social Media Comments
## 4 Model Evaluation on Croatian Social Media Dataset
### Training the Classification Models  
To train the classification models for disinformation detection, we used three different machine learning algorithms: a Support Vector Machine (SVM), a Random Forest, and a neural network. These models were selected due to their proven effectiveness in text classification tasks, with the SVM excelling in high-dimensional spaces, the Random Forest offering robust performance and interpretability, and neural networks providing the ability to capture complex, non-linear patterns in the data. The SVM model was trained with a Radial Basis Function (RBF) kernel to account for non-linear relationships in the data. For the Random Forest model, an ensemble of decision trees was used to improve classification accuracy and robustness against overfitting. The neural network model was designed with a single hidden layer of 10 neurons and optimized with the stochastic gradient descent (SGD) algorithm to efficiently minimize the loss function. Each model was trained on a labeled dataset of social media posts, extracting features from the text data, such as word embeddings and sentiment scores. The training process involved splitting the dataset into a training set and a validation set, tuning the hyperparameters through cross-validation, and evaluating the model performance using metrics such as accuracy, precision, recall, and F1 score. This comprehensive approach ensured that our models were well equipped to effectively identify and classify disinformation. Figure 5a-c show the confusion matrix for each model. The results are also shown in Table 1. From the data displayed in the confusion matrix, we can calculate the metric performance parameters for every learning mechanism that we used for our model. Despite employing relatively simple methods, this approach involved extensive pre-training of the model over multiple epochs using the annotated dataset. This rigorous training process enabled the model to discern subtle differences in intent and sentiment, thereby enhancing the accuracy of hate speech detection. Figure 4: Rand Index over number of dimensions. The Support Vector Machine (SVM) model achieved a classification accuracy of 78.05%, indicating that it correctly identified disinformation in 78.05% of the cases. The high precision of 84.13% indicates that the model effectively identifies positive instances with minimal false alarms. However, the recall of 67.09% suggests that the model misses a notable proportion of actual positive cases. The F1 score of 74.62% confirms a reasonable balance between precision and recall, making the model suitable for applications where both false positives and false negatives have significant implications. This result suggests that the SVM model is effective in distinguishing between disinformation and legitimate content, though there is still room for improvement. The accuracy reflects the model's ability to generalize from the training data to unseen data, which is crucial for real-world applications. However, to further enhance performance, it may be beneficial to explore additional feature engineering techniques, optimize hyperparameters, or consider combining an SVM with other models in an ensemble approach. Overall, the SVM's performance is promising and provides a solid foundation for detecting disinformation on social media. The Random Forest model achieved an overall accuracy of 76.36%, indicating a reasonable balance between correct and incorrect predictions. The precision of 78.57% suggests that most of the positive predictions are accurate, reducing the likelihood of false alarms. However, the recall of 69.62% reveals that the model misses a significant proportion of actual positive cases, which could be critical depending on the application. The F1 score of 73.84% underscores a moderate balance between precision and recall, making the model suitable for contexts where both false positives and false negatives are consequential.