# Authorship Attribution on Short Texts in the Slovenian Language
## 3 Materials and Methods
### Model Training and Evaluation  
To create a language model that solves the problem of AA, our approach was to fine-tune an existing language model. [PERSON] et al. [11] demonstrated that fine-tuning BERT is a feasible approach, so we focused on this family of language models only.  
BERT is a language model based on the Transformer architecture [9; 28]. Built using the attention mechanism, it uses an encoder and decoder to create a bidirectional representation of the input tokens. Due to its structure, a pre-trained model can be very useful for fine-tuning further NLP tasks without having to create custom networks from scratch.  
First, we decided to use a monolingual model to achieve the best performance since monolingual models have been shown to perform better in various NLP tasks than multilingual models [29; 30]. Since the texts in our dataset are written entirely in Slovenian, the base model we fine-tuned was a Slovenian monolingual model SloBERTa 2.0 [8]. SloBERTa is the first monolingual Slovenian language model. Its architecture is the same as that of RoBERTa, a model similar in design to BERT, while the tokenization model was inspired by CamemBERT [31; 32].  
To see the effect of using a multilingual model on our task and to compare the results, we also fine-tuned the mBERT model, which supports Slovenian and 103 other languages [9]. We fine-tuned those two models for each of the created datasets.  
The training was conducted in ten epochs. The authors of the BERT's original paper [9] recommend that the training is performed on at least five epochs for a wide variety of NLP tasks. For the AdamW optimizer, we set the weight decay parameter to 0.01 and the learning rate parameter to \(2\cdot 10^{-5}\). This value was chosen to avoid the problem of catastrophic forgetting [33]. Setting this value too high would cause the model to forget previously learned knowledge as new inputs are introduced during the training phase. Since our problem is a single-label classification problem, we chose to use the cross-entropy loss during training [34]. The full list of hyperparameters is described in Table 2. To perform the fine-tuning, we used the Hugging Face trainer [35; 36].  
Separately, we also fine-tuned mBERT on the IMDb1m dataset. We did not repeat the procedure with SloBERTa as the base model because all the posts in the dataset are in English.  
Figure 2: The distribution of the number of comments created by 200 of the most active users from the IMDb1m dataset. Users are sorted in descending order by the number of comments they posted. The x-axis denotes the ID of the user.  
To fine-tune the models, we used the Hugging Face Transformers Python library [35] on a system with an NVidia V100 PCIe (24 GB GDDR5 memory) graphics card, which took 2-14 h per model, depending on the size of the dataset.  
After the training of each model was completed, we calculated the macro-averaged F1 scores. We also evaluated the models using out-of-class comments, which allowed us to gain insight into the potential performance drop when dealing with the open-class problem type.  
To calculate the F1 score, we used the following formula:  
\[\text{F1 score}=2\times\frac{\text{precision}\times\text{recall}}{\text{ precision}+\text{recall}}\]  
Here, we used the following formulas for precision and recall:  
\[\text{precision}=\frac{TP}{TP+FP}\quad\text{and}\quad\text{recall}=\frac{TP}{ TP+FN}\]  
\(TP\) means true positive, \(TN\) means true negative, \(FP\) means false positive, and \(FN\) means false negative.  
Then, to calculate the macro-averaged F1 score, we used the following formula:  
\[\text{Macro-averaged F1 score}=\frac{\text{sum of F1 scores of all classes}}{\text{number of classes}}\]  
For the models fine-tuned on the IMDb1m dataset, our goal is to compare our approach to the one proposed by [PERSON] et al. [11]. In their work, they calculated the accuracy of each model, so we did the same for the relevant models.  
To calculate the accuracy, we used the following formula:  
\[\text{Accuracy}=\frac{TP+TN}{TP+FP+TN+FN}\]