## 6 Experiments and Results
Figure 1: Macro-average F1 score values (with confidence intervals) achieved on the MM18x58 and MM18x58_En datasets in multilingual and crosslingual experiments, respectively. To determine the most accurate FFNN-based binary classifier, various feature types were investigated (the results are summarized in Table 3):  
* _LaBSE vectors._
* _Softmax values_ for each of the 18 known classes were obtained from the multi-class classification model (specifically, LaBSE+FFNN, which achieved the best results with the MM18x58 dataset) and were used as the feature vector. * _Penultimate layer's values_ are taken from the same model that provides the _softmax values_, but are extracted before the application of the softmax function. * _Cosine similarities_ to the cluster centers of all 18 known classes were used as feature vectors. These cluster centers were calculated by averaging all LaBSE instance vectors belonging to each of these classes. * Concatenation (denoted as \"+\") of various features. The best results, characterized by the lowest loss function values, were achieved using the LaBSE+FFNN approach (see Table 3); hence, this approach was utilized in subsequent experiments. During these experiments, we (1) removed class X from both the training and validation splits; (2) retrained the multi-class LaBSE+FFNN classification model using the MM18x58 dataset, excluding class X; (3) retrained the binary LaBSE+FFNN classification model using the MM18x58_binary dataset, also excluding class X; (4) evaluated the method's ability to detect the new class by using all instances of class X across training, validation, and testing splits. The results obtained are summarized in Table 4.  
The accuracy values for classifying the removed class X as the _new_ class. \begin{table}
\begin{tabular}{l c c} \hline \hline
**Class X** & **Num. of Tested Instances** & **Accuracy** \\ \hline
1. AlternativeEnergy & 43 & 1.000 \\
2. ClimateAction & 47 & 1.000 \\
3. ClimateChange & 1317 & 0.992 \\
4. CoronavirusInfection & 274 & 0.971 \\
5. CybersecurityAntifraud & 302 & 0.990 \\
6. Drugs & 96 & 0.906 \\
7. EUEconomy & 79 & 0.941 \\
8. EUInternet & 54 & 1.000 \\
9. EnergyMarketsandStrategies & 17 & 1.000 \\
10. EuropeanCouncil & 392 & 0.980 \\
11. Europol & 834 & 0.978 \\
12. FightagainstFraud & 83 & 0.940 \\
13. FinancialEconomicCrime & 30 & 1.000 \\
14. ForgeryMoney & 9 & 1.000 \\
15. InformationSecurity & 113 & 1.000 \\
16. Migration & 181 & 0.972 \\
17. NATO & 10,796 & 0.796 \\
18. TerroristAttack & 334 & 0.997 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: The accuracy values for classifying the removed class X as the _new_ class. \begin{table}
\begin{tabular}{l c} \hline \hline
**Features** & **Avg. Loss \(\pm\) Confidence Int.** \\ \hline LaBSE vectors & 0.0017 \(\pm\) 0.0002 \\
1. Softmax values & 0.0346 \(\pm\) 0.0014 \\
2. Penultimate layer’s values & 0.1101 \(\pm\) 0.0596 \\
3. Cosine similarities & 0.0049 \(\pm\) 0.0077 \\
1 + 2 & 0.0661 \(\pm\) 0.0850 \\
1 + 3 & 0.0035 \(\pm\) 0.0031 \\
2 + 3 & 0.0137 \(\pm\) 0.0334 \\
1 + 2 + 3 & 0.0057 \(\pm\) 0.0036 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: The loss values, averaged from 5 runs and presented with confidence intervals, were obtained by exploring different features with the optimized binary FFNN. The “+” symbol denotes the concatenation of features.