## 6 Experiments and Results  
During the investigation, we conducted closed-set multilingual and crosslingual text classification experiments using the MM18x58 and MM18x58_En datasets, respectively. These experiments were designed to compare the approaches presented in Section 5. For the training of classification models with BERT+CNN, LaBSE+FFNN, LaBSE_fine_tuning, and XLM-R_fine_tuning, we set a maximum of 100 epochs but used early stopping to monitor the validation loss. Early stopping was triggered when the loss showed minimal improvement (with a minimum delta of 0.01) and after a patience of three epochs without improvement. All other parameters remained at their defaults. We also used default parameters with Davinci_fine_tuning. We repeated each experiment five times, averaged the results, and calculated the confidence intervals. Because all memory-based approaches (LaBSE_LangChain_k1, LaBSE_LangChain_k10, ADA_LangChain_k1, and ADA_LangChain_k10) do not require training but rather the storage of training instances, the validation dataset was not utilized. Since there is no randomness with memory-based approaches, each experiment was performed only once; neither averaging of results nor confidence intervals were needed. Our closed-set text classification experiments were conducted with a highly imbalanced dataset, with the largest class, _NATO_, representing 75%, 69%, and 72% of all instances in the training, validation, and testing splits, respectively. To address the potential issue of the accuracy metric favoring the majority class and potentially yielding misleadingly high scores, we have chosen the F1 score (see Equation (1)) as our primary evaluation metric. Furthermore, we adopted the macro-averaged evaluation method to ensure unbiased evaluation across all classes. The results with different approaches are summarized in Figure 1.  
\[F1=\frac{2\cdot TP}{2\cdot TP+FP+FN} \tag{1}\]  
where \(TP\) (True Positives) represents the number of instances correctly predicted as the class of interest, \(FP\) (False Positives) represents the number of instances incorrectly classified as the class of interest, and \(FN\) (False Negatives) represents the number of instances that are actually the class of interest but were predicted as any other class. To determine whether the differences between the results achieved with different methods are statistically significant, we used the Student's \(t\)-test [41] with a significance level of \(\alpha=0.05\). In cases where we needed to compare a group of values in one experiment with a single value in another, we conducted the one-sample \(t\)-test [42] with the same \(\alpha=0.05\). The second set of experiments with the binary dataset (see Table 2) was dedicated to finding the best set of features that could effectively distinguish _new_ topics from _known_ ones, addressing the open-set text classification problem. We tested the optimized FFNN approach using the sigmoid function instead of softmax for these experiments. For details on the optimization of FFNN with Hyperas and Hyperopt, see the LaBSE+FFNN description in Section 5. Once the probabilities were obtained via the sigmoid function, binary cross-entropy was used as the loss function \(L\) presented in Equation (2). As the evaluation metric, this loss function is much more precise (especially in scenarios with a very imbalanced dataset, as in our case) because it measures how far off the predictions are from the actual values. The accuracy defined in Equation (3) was used as our second evaluation metric. \[L=-(y\log(p)+(1-y)\log(1-p)) \tag{2}\]  
where \(y\) is the actual label, which can be either 0 or 1; \(p\) is the predicted probability, the output from the sigmoid function, that the observation is of class 1. \[Accuracy=\frac{TP+TN}{TP+TN+FP+FN} \tag{3}\]  
where \(TN\) (True Negatives) represents the number of instances that are not the class of interest and were correctly not predicted as the class of interest. For the other notation, see Equation (1).