## 5 Modeling
### Train Process  
In this sub-section, the authors review the training process, the parameters applied, and the metrics used.  
First, the dataset was split into the train, validation, and test parts with a fraction of 85%. In addition, the authors applied oversampling of low classes with a threshold value set to 500 and set up a test dataset to include only single-labeled elements. Classes distribution for each part can be found in Appendix A. The dataset was batched by 64 elements in a batch. Considering that the target variables are encoded as binary values, the loss function was set to binary cross entropy [23].  
The authors decided to use the AdamW optimizer because in the context of emotional analysis of text data, the use of the AdamW optimizer can be particularly useful as it helps to prevent overfitting, which is a common problem in natural language processing tasks. By using this optimizer, the model is better able to generalize to new, unseen data, improving its overall performance and making it more useful for real-world applications [25]. This is the optimizer BERT was initially trained on, and it performs regularization by weight decay. The authors applied a similar training schedule as BERT pre-training: linear decay of a notional initial learning rate, prefixed with a linear warmup phase over the first 10% of training steps [6]. The training was performed up to 10 epochs. As a result, each epoch consisted of 463 steps, adding to 4630 steps in total with 463 warmup steps. The initial learning rate was set to \(3\times 10^{-4}\). Decreasing the learning rate should perform better [26]; however, it would also require additional computation time or additional resources. Furthermore, the learning rate value has a higher impact on the model performance than the batch size [26].  
During the model training and testing, the authors tracked the precision and recall for each predicted class individually and for all classes.  
The indicator precision describes how close the measurement values are to each other. Precision is the number of true positives (TP) over the number of true positives (TP) plus the number of false positives (FP).  
\[Precision=\frac{TP}{(TP+FP)}\]  
TP--True Positive, FP--False Positive  
Recall defines how accurately our model is able to identify the relevant data. Recall is the number of true positives (TP) over the number of true positives (TP) plus the number of false negatives (FN)  
\[Recall=\frac{TP}{(TP+FN)}\]  
FN--False Negative  
Additionally, the authors tracked the F1-Score of all classes with weighted, micro, and macro averaging. A threshold value for all metrics was set to 0.5. The F1-Score is the Harmonic mean of the Precision and Recall.  
\[F1=\frac{2\times(Precision\times Recall)}{(Precision+Recall)}\]  
Finally, the authors initialized the training early stopping to stop the training process in case validation loss does not decrease during the last two epochs. Thus, the model was trained as long as it increased its performance. During experiments, training was stopped after 7 epochs.  
Figure 3 shows how precision, recall, and F1-Score metrics, as well as the value of the loss function, changed over epochs on the train data during the model fit.  
Similarly, changes in the values of the same metrics and the loss function on the validation dataset are presented in Figure 4.  
As we can see, the value of the loss function decreased while all of the F1-Score metrics increased with each epoch. Thereafter, the model gained performance and did not tend to overfit.  
Finally, the authors ran the same metrics on the test dataset, which consists of the data that the model has not seen before, and analyzed the results. Having conducted a series of experiments, the authors present the best results in Section 6 of this study.  
Figure 3: Change in values of the loss function, Precision, Recall, and F1-Score with micro, macro, and weighted averaging for all classes on the training dataset over train epochs (source: author’s development).