# Creation, Analysis and Evaluation of AnnoMI, a Dataset of Expert-Annotated Counselling Dialogues +
## 8 Topic-Specific and Cross-Topic Performance
### Topic-Specific Performance  
To obtain the performance on topic \(T_{i}\), we re-use the 5-fold CV models for the two tasks (Section 7), but we test each model only on a \(T_{i}\)-specific subset of the corresponding test fold. Specifically, the subset consists entirely of utterances that are originally from conversations of topic \(T_{i}\). By averaging the performance of the 5 models on their respective \(T_{i}\)-specific test-fold subsets, this method covers all \(T_{i}\) utterances and thus yields a reliable measure of the \(T_{i}\)-specific performance of each model type.  
Generally, it is clear that the model performance, especially that of the BERT variants, follows the topic-wise ordering below:  
* **Therapist Behaviour Prediction:** reducing alcohol consumption \(>\) smoking cessation \(>\) reducing recidivism;
* **Client Talk Type Prediction:** reducing alcohol consumption \(\approx\) smoking cessation \(>\) reducing recidivism.  
One contributing factor to the performance gaps between different topics could be topic coverage, namely the number of utterances from sessions of a particular topic, as better coverage entails more data used for training. For example, reducing alcohol consump  
\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Result Format** & \multicolumn{3}{c}{**Topic-Specific \(\rightarrow\) Cross-Topic**} \\ \hline
**Topic** & **Reducing Alcohol Consumption** & **Reducing Recidivism** & **Smoking Cessation** \\ \hline \hline \multicolumn{4}{c}{**(Main) Therapist Behaviour Prediction**} \\ \hline BERT w/ Adapters & \(0.74\to 0.74\) & \(0.63\to 0.62\) & \(0.70\to 0.72\) \\ BERT w/o Adapters & \(0.72\to 0.75\) & \(0.65\to 0.66\) & \(0.72\to 0.70\) \\ CNN & \(0.59\to 0.55\) & \(0.50\to 0.52\) & \(0.64\to 0.60\) \\ Random Forest & \(0.49\to 0.49\) & \(0.40\to 0.36\) & \(0.53\to 0.48\) \\ \hline \multicolumn{4}{c}{**Client Talk Type Prediction**} \\ \hline BERT w/ Adapters & \(0.55\to 0.52\) & \(0.41\to 0.43\) & \(0.56\to 0.51\) \\ BERT w/o Adapters & \(0.54\to 0.52\) & \(0.41\to 0.42\) & \(0.55\to 0.50\) \\ CNN & \(0.47\to 0.45\) & \(0.39\to 0.39\) & \(0.50\to 0.43\) \\ Random Forest & \(0.42\to 0.38\) & \(0.33\to 0.34\) & \(0.37\to 0.32\) \\ \hline \hline \end{tabular}
\end{table}
Table 17: Topic-specific and cross-topic performance (macro F1) in the Original Unbalanced setting for (1) therapist behaviour prediction and (2) client talk type prediction. The 3 topics are reducing alcohol consumption, reducing recidivism, and smoking cessation. \(\uparrow\)/\(\downarrow\): cross-topic performance is higher/lower than topic-specific performance.  
tion has more utterances than reducing recidivism (Table 3) and correspondingly also better performance.  
However, it is also clear that the performance on reducing recidivism conversations is considerably lower than on smoking cessation, despite the slightly larger coverage of reducing recidivism. This is more likely because the utterances of the topic themselves are more semantically challenging for the task, and it also shows the necessity to include a wide range of topics in a counselling dialogue dataset.