# Creation, Analysis and Evaluation of AnnoMI, a Dataset of Expert-Annotated Counselling Dialogues +
## 7 Utterance-Level Prediction Experiments
### Task 2: Client Talk Type Prediction  
Client talk type prediction aims to produce the correct client Talk Type label given a client utterance. As shown in Table 16, this task records universally lower scores than Task 1 for all the trained models--the best BERT-variant performance scores are around 0.55 for macro F1, while CNN and Random Forest score less than 0.47, irrespective of data augmentation. Two factors likely responsible for the performance gap between the two tasks are dialogue context and annotation noise. In some cases, the talk type of a client utterance can only be determined with context grounding. For example, \"Yeah\" as a reply to \"So you work out every day?\" is a neutral talk, but it should be Change Talk when it follows \"Don't you ever wish things were different?\". Moreover, the IAA (Fleiss' kappa) for client talk type is around 0.47, while it is 0.74 for therapist behaviour, which suggests that annotating the talk type is more challenging and therefore more noise is present in the labelling. Inevitably, such noise makes it more difficult to optimise the trainable models.  
Among the talk types, Neutral Talk has the best performance, followed by Change Talk and Sustain Talk, which matches the class distribution (Table 14), similar to the finding in Task 1 (Section 7.1). Interestingly, in some cases, the inter-class performance gap is reduced thanks to data augmentation. For example, the gap between Change Talk and Neutral Talk is 0.23 F1 in Original Unbalanced but 0.14 F1 in Augmented Balanced, even though most of it is attributed to the performance decrease of 0.07 F1 on Neutral Talk. Unsurprisingly, both Change Talk and Sustain Talk are frequently misclassified as Neutral Talk, even by the best-performing model, BERT w/ Adapters, as can be seen in the confusion matrix (Figure 13). Using dialogue context as an additional input may reduce misclassification to a certain extent, as hypothesised before, but class imbalance may ultimately become the bottleneck for performance improvement [12]. We leave further probing to future work.  
\begin{table}
\begin{tabular}{l c c c c} \hline \hline \multicolumn{1}{c}{**Result Format**} & \multicolumn{4}{c}{**Original Unbalanced (Augmented Balanced)**} \\ \hline \multicolumn{4}{c}{**Client Talk Type Prediction**} \\ \hline
**Model** & **F1-Macro** & **F1-Change Talk** & **F1-Neutral Talk** & **F1-Sustain Talk** \\ \hline BERT w/ Adapters & 0.55 (0.53\(\downarrow\)) & 0.51 (0.53\(\uparrow\)) & 0.74 (0.67\(\downarrow\)) & 0.39 (0.37\(\downarrow\)) \\ BERT w/ Adapters & 0.53 (0.52\(\downarrow\)) & 0.49 (0.51\(\uparrow\)) & 0.71 (0.67\(\downarrow\)) & 0.39 (0.39) \\ CNN & 0.47 (0.46\(\downarrow\)) & 0.45 (0.44\(\downarrow\)) & 0.65 (0.63\(\downarrow\)) & 0.31 (0.31) \\ Random Forest & 0.39 (0.44\(\uparrow\)) & 0.38 (0.40\(\uparrow\)) & 0.71 (0.65\(\downarrow\)) & 0.10 (0.26\(\uparrow\)) \\ Prior & 0.33 (0.31\(\downarrow\)) & 0.29 (0.31\(\uparrow\)) & 0.57 (0.42\(\downarrow\)) & 0.14 (0.20\(\uparrow\)) \\ Uniform & 0.31 (0.31) & 0.31 (0.31) & 0.42 (0.42) & 0.20 (0.20) \\ \hline \hline \end{tabular}
\end{table}
Table 16: Macro F1 and per-class F1 scores of client talk type prediction. All results averaged from 5-fold cross-validation. \(\uparrow\)/\(\downarrow\): performance increase/decrease by using Augmented Balanced compared to using Original Unbalanced.  
Figure 13: Confusion matrix of BERT w/ Adapters for client talk type prediction in the Original Unbalanced setting. Normalised by row.  
**Overall Remarks:** We establish the single-utterance BERT variants as baselines for both tasks to facilitate comparison with more advanced models in future work. For example, one possibility for improvement would be to formulate Tasks 1 and 2 as BERT-style next-sentence prediction tasks, using the dialogue context as the first sequence and the therapist/client utterance as the second. This setup would enable more contextualised prediction, especially for client talk type prediction.