## 5 Results
### Zero-Shot Topic Classification
#### 5.1.2 New Results on SGC5--Topic and Subtopic Level
As can be seen at the topic level, the classification performed by the BART model is significantly better than the model provided by the CoBot topic classifier. In 101 cases, both models agreed (tied) in their annotation and were right, and only in 14 were they both wrong. However, in 82 cases, the topic classified by BART was better than that of the label provided by the CoBot topic classifier, whereas the CoBot topic classifier was right in 3 cases ahead of BART. In more detail, we found that the CoBot topic classifier labeled turns as \"other\" topic in 71 cases. In 58 out of those 71 cases, the human label was 2; that is, the topic annotated by BART provided a better label (i.e., different from \"other\"); in the remaining cases, the human label was \(-1\); that is, none of the models were correct in predicting the topic. These results mainly show that the proposed approach with a higher number of topics is able to successfully break down the turns annotated as \"other\" by the CoBot topic classifier, and therefore provides a better classification to the annotator. Figure 4: Confusion matrix of the Zero-Shot BART model versus the CoBot topic classifier on the SGC5 data. The percentage and number of matches for each topic are shown. Each time the BART model labels a turn, it returns all the possible labels it has been asked to use to classify, along with their respective associated score. The scores correspond to the probability that the label is true (confidence). During the classification of all turns, the label with the highest associated score was taken, regardless of any threshold. Figure 5 shows the number of different topics annotated by BART and their average score associated with each topic. As can be seen, only 13 of the 18 topics were detected in the selected random subset. The missing topics are: news, photography, vehicle, weather, and other. The most represented topics were food, music, and movies. The topics with less confidence were family, finance, and education. It should be noted that \"family\" had a large number of occurrences; however, its average score associated was low. Regarding finance and education, they had low representation and a very low average score associated, making these the worst topics to be classified by BART. Taking only the number of hits of the BART classifier at the topic level into account, we calculated how many of these hits have an associated score above a minimum threshold of 0.9 (as in Figure 3). In \(\sim\)71% of the cases it was favorable, while in \(\sim\)29% it was lower, thus demonstrating that the confidence level of the classifier is high. At the subtopic level, the proposed BART labels were correct with an accuracy of 71.5%. This result shows that the fine-grained Zero-Shot annotations are suitable, coherent, and scalable. Of the 143 cases in which the subtopic was correct, in 142, the topic was also labeled as correct. The subtopics that were repeated the most throughout the subset were friends (family), genre (movies/music), healthy (food), and song (music). For the hit cases, only \(\sim\)31% exceeded the threshold of 0.9, showing less confidence in the choices made, although they were correct in a high percentage. \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Scores**} & \multicolumn{2}{c}{**Topic Level**} & \multicolumn{2}{c}{**Subtopic Level**} \\  & **Hits** & **Accuracy (\%)** & **Hits** & **Accuracy (\%)** \\ \hline
2 & 82/200 & 41 & - & - \\
1 & 3/200 & 1.5 & 143/200 & 71.5 \\
0 & 101/200 & 50.5 & 57/200 & 28.5 \\ \(-1\) & 14/200 & 7 & - & \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Results of manual annotations conducted at the topic and subtopic level on SGC5 data, with BART model versus CoBot topic classifier. Figure 5: Topic results in the SGC5 subset of 200 dialogue turns. Upper graph: number of topics annotated by the BART classifier. Lower graph: average score associated with each topic classified by the BART model.