# Detecting Disinformation in Croatian Social Media Comments
## 4 Model Evaluation on Croatian Social Media Dataset
### Features of the Dataset Used  
We can observe that the second comment represents a problem, because it is a null vector, meaning it does not contain words that have representation in the defined vector space. Following a closer examination of the input dataset, we decided that all sentences resulting in null vectors would be ignored. The reasoning for such a decision is that there are many sentences that have incorrect grammar (wrong spelling or using incorrect diacritic symbols specific to the Croatian language--diacritic marks on letters like \(\xi\), \(\xi\), \(\bar{s}\), \(\bar{z}\), and d are omitted) and in the process of vectorization, written words in those comments would not be recognized as grammatically correct words. By implementing this approach, approximately 1400 comments were excluded from the dataset as null vectors. The exclusion of these null vectors from the final analysis was crucial to preserving the integrity and accuracy of the data. This ensured that only comments with grammatically correct and recognized words were included in the vectorization and subsequent analyses. For our research, we used the offensive language dataset of Croatian, English, and Slovenian comments, FRENK 1.1 [52]. The dataset consists of public Facebook comments published on Croatian platforms. The FRENK datasets provide a crucial resource for analyzing socially unacceptable discourse (SUD) on Facebook, with a specific focus on remarks about immigrants and LGBTQ+ individuals, chosen due to the high prevalence and significant social impact of hate speech directed at these groups. The Croatian dataset, part of the broader FRENKproject, was created by analyzing comments on Facebook posts from mainstream media sources. The selection criteria ensured that the data were representative and comparable across different languages and contexts. Posts related to migrants and LGBTQ+ topics were identified through a combination of manual and automated methods, resulting in a robust training set that accurately represents these issues. The public comments were annotated for different types of SUD and their targets. Key characteristics of the dataset include the following:  
* The average text length is 218 characters--comments vary in length, providing a comprehensive overview of different discourse styles. * Annotations include six types of SUDs, such as background violence and offensive speech, and five targets, including migrants, LGBTQ+ individuals, and journalists. * The average number of annotations is 8.59. Since there are very few available Croatian language-based datasets that could have been used for this research, and this research focused on Facebook remarks about immigrants and LGBTQ+ individuals, we decided to train our model on this dataset. The rationale lies in the observed trend of hate speech and disinformation targeting these vulnerable groups. Disinformation often fuels hate speech by spreading false or misleading information that incites fear, prejudice, and hostility. By analyzing the discourse on these topics, the dataset aims to uncover patterns of disinformation that contribute to the spread of hate speech, ultimately informing strategies to counteract both phenomena. While the FRENK 1.1 dataset offers a valuable resource for disinformation detection in the Croatian language, its thematic focus on migrants and LGBTQ+ topics inherently constrains the generalizability of our findings. The dataset captures disinformation strategies and narratives within these specific domains, which may not comprehensively reflect the linguistic patterns or rhetorical devices used in disinformation targeting other social, political, or economic issues. Consequently, the performance of the proposed models might vary when applied to datasets encompassing a broader array of topics.