## 5 Results
### Zero-Shot Topic Classification
#### 5.1.2 New Results on SGC5--Topic and Subtopic Level  
**BART vs CoBot topic classifiers:** As commented in Section 4.1, the SGC5 data were automatically labeled (silver labels) using a sample pre-trained topic classifier provided to the SGC5 teams as part of the CoBot architecture and then labeled with four NLI-based models (Deberta-v3, Deberta-v3-large, Bart-large, and Distillbert-uncased). These four models were tested to produce the same 11 topics provided by the CoBot classifier (Table 2). They also received the same amount of dialogue context, i.e., the current turn of the dialogue and the last three pairs of turns in the history. The goal here is to detect which of the four untrained Zero-Shot models provides the closest results when using the silver labels and then select that model for the subtopic level classification. Table 5 shows the results of the four models on the silver labels at topic level. The model with the best performance was BART-large-mnli with an F1 Score of 0.26%; therefore, this model was selected to be used for the rest of the experiments. The confusion matrix between the CoBot topic classifier and BART is shown in Figure 4.  
The confusion matrix shows that there is a high coincidence for sports, politics, movies, books, and music topics, while there is less coincidence for science, general, phatic, interactive, inappropriate, and other topics. This is probably due to the overlap in the data used to train the MNLI and the one used for testing. Moreover, topics with high matching are generally easier to classify, while the low matching topics are too open or prone to more subjective annotations. **Topic and Subtopic detection:** To validate the possibility of performing automatic detection of topics and subtopics using a Zero-Shot approach (Section 3.1), the SGC5 data were automatically annotated at the topic and subtopic level using the BART model. In this case, the number of topics for BART were the 18 proposed as an improvement (Table 1), while for the CoBot topic classifier, they were the same 11 that the classifier was able to identify (Table 2). The common topics between the BART and CoBot topic classifiers are: sports, movies, books, music, science, and other. The unique topics for the BART classifier are: animals, video games, politics, art, teacher, family, finance, cars, astronomy, food, photography, newspaper, fashion, climate conditions, and Facebook. As for the different topics that the CoBot topic classifier is able to detect, they are: general, phatic, interactive, and inappropriate. To evaluate the real quality of automatic classifications performed using BART compared to those provided by the CoBot topic classifier, three experts manually evaluated 200 dialogue turns. These annotations were divided into two levels. **Topic level:** the topics provided by the BART and CoBot topic classifiers were annotated according to the following labels. * 2: BART topic is correct. * 1: CoBot topic is correct. * 0: Both BART and CoBot topics are correct. \begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Model** & **Accuracy (\%)** & **F1 Score** & **Precision** & **Recall** \\ \hline cross-encoder/nli-deberta-v3-base & 10.08 & 0.08 & 0.22 & 0.15 \\ cross-encoder/nli-deberta-v3-large & 7.55 & 0.06 & 0.18 & 0.15 \\ facebook/bart-large-mnli & 25.98 & 0.26 & 0.30 & 0.29 \\ typeform/distillbert-base-uncased-mnli & 0.83 & 0.01 & 0.09 & 0.02 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: NLI topic classifier models vs. CoBot topic classifier. The best F1-Score result among all models is shown underlined. * \(-1\): Neither the BART nor the CoBot topics are correct. **Subtopic level:** as an extension of the topic level, we annotated the same 200 random dialogues also at the subtopic level using the following labels:  
* 1: The BART subtopic is correct. * 0: The BART subtopic is not correct. Table 6 shows the results of the two annotation levels.