# Detecting Disinformation in Croatian Social Media Comments
## 4 Model Evaluation on Croatian Social Media Dataset  
The TF-IDF algorithm receives an array of words as input; each word is called a token. Due to the nature of the grammar of the Croatian language (i.e., there are many grammatical rules and cases and, consequently, many different forms of the same word with the same meaning), we have adapted the tokenization of the input using a classla library [49], a fork of the official Stanford NLP Python library [50], to obtain the word roots.  
The corpus is available in vertical format, compatible with Sketch Engine and CWB concordances, and provides information at the text, paragraph, sentence, and token levels:  
* Each text is accompanied by metadata including text id, title, URL, domain, top-level domain (e.g., \"com\"), and predicted genre category.
* Paragraphs are annotated with paragraph id, the automatically identified language, and paragraph quality, with labels such as \"short\" or \"good\" assigned based on length, URL, and stopword [51].
* Sentences within paragraphs have their sentence id as metadata.
* Tokens are provided in tabular format with their linguistic annotation.  
In our adaptation of the tokenization process, randomly selected words from the text are replaced with a [MASK] token, which serves as a learnable parameter. The model subsequently attempts to predict the index of the word from the dictionary that corresponds to the position of the [MASK] token. This approach enables the network to discern dependencies between the masked word and its surrounding context. Following the pre-training phase, the model undergoes fine-tuning on a specific dataset to address tasks, such as question answering or text classification. Tokenization is pivotal in this process, as it segments the text into manageable units, thereby facilitating effective processing by the model.  
The following example demonstrates the principle used in the presented approach. Suppose that the dataset is represented by the following three comments:* \"Danas idemo na pice\" (in English: We are going to drink today).
* \"Sutra ce biti bole\" (in English: It will be better tomorrow).
* \"Svaki dan je novi pocetak\" (in English: Every day is the new beginning). In the first step, the TF-IDF analysis determines which words will represent the axes in vector space. After that, in the second step, the vector representations are created by counting the number of words that appear in comments. So, using the example, the output will be the following: Axis: [\"dan\", \"danas\"] Representation: [[0. 1.], [0. 0.], [1. 0.]]. This matrix indicates the TF-IDF values for each term in each document:
* The first comment, \"Danas idemo na pice.\", has a TF-IDF value of 1 for \"danas\" and 0 for \"dan\". This suggests that \"danas\" is a significant term in this comment.
* The second comment, \"Sutra ce biti bole.\", has TF-IDF values of 0 for both \"danas\" and \"dan\", indicating that neither term is significant in this comment.
* The third comment, \"Svaki dan je novi pocetak.\", has a TF-IDF value of 1 for \"dan\" and 0 for \"danas\". This highlights \"dan\" as a significant term in this comment. We used a basic spelling checker to address the grammatical inaccuracies found in the dataset, with a particular focus on the omission of Croatian diacritic marks (\(\xi\), \(\xi\), \(\xi\), \(\bar{z}\), and d) [44]. The spell-checking process was applied to identify comments with missing diacritical marks, ensuring the text's orthographic accuracy. Incorporating this spelling correction step before vectorization improved the text representation quality, facilitating precise word identification and processing. This approach maintained the dataset's linguistic integrity, enabling subsequent analyses to be conducted on grammatically corrected text data. Consequently, the erroneous comments were marked as null vectors, which led to more reliable and valid outcomes. By implementing this approach, approximately 1400 comments were excluded from the dataset as null vectors.