# Creation, Analysis and Evaluation of AnnoMI, a Dataset of Expert-Annotated Counselling Dialogues +
## 7 Utterance-Level Prediction Experiments  
From the annotation labels, various utterance-level prediction tasks can be readily defined. In this section, we focus on two tasks: therapist behaviour prediction and client talk type prediction. We introduce these tasks as examples of potential real-world applications of AnnoMI, in order to inspire future tasks based on the dataset. From a practical point of view, an accurate prediction model of therapist behaviour and client talk type can automatically label utterances and thus facilitate therapy quality monitoring and provide feedback for the therapist (Section 2.1), ultimately improving counselling quality. In experimenting with these tasks, we also examine the impact of the relatively lower IAAs of some utterance attributes on the prediction performance. Figure 11: t-SNE of client utterance embeddings. Figure 10: t-SNE of therapist utterance embeddings. While imbalance exists between the high- and low-quality dialogue volumes, we expect its impact on the tasks to be minor, since they are not related to MI quality directly. For future work exploring session- or utterance-level MI quality classification, however, remedies such as data augmentation will be needed to address the imbalance. Each task allows a single utterance as the input and requires a class label as the output. We experiment with 4 machine learning models, as listed below. We implement the BERT variants with AdapterHub [37; 38] ([[https://github.com/Adapter-Hub/adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers)]([https://github.com/Adapter-Hub/adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers)), accessed on 14 February 2023), the CNN models with Keras ([[https://keras.io/](https://keras.io/)]([https://keras.io/](https://keras.io/)), accessed on 14 February 2023), and the other models with Scikit-learn [39]. * **BERT w/o Adapters**: BERT-base-uncased [40] fine-tuned on AnnoMI. * **BERT w/ Adapters**: BERT-base-uncased with adapters [37; 41] fine-tuned on AnnoMI. Adapters are a small set of task-specific parameters that can be easily plugged into transformer [42] models, so that only the lightweight adapters are updated during fine-tuning, while the rest of the model is frozen. * **CNN**: convolutional neural networks initialised with word2 vec embeddings [43] and fine-tuned on AnnoMI. * **Random Forest**: random forest with tf-idf features. We also use 2 random baseline classifiers for comparison:  
* **Prior**: random prediction based on the class distribution in the training set;
* **Uniform**: random prediction based on the uniform distribution of the classes. Since duplicate utterances are present in AnnoMI, especially in the categories of Other and Neutral Talk (e.g., \"Uh-huh\" and \"OK\"), we perform de-duplication as a preprocessing step. Specifically, if multiple identical utterances have the same label, we randomly select one of them to keep and remove the others. The distribution of (Main) Behaviour and Talk Type after this step is shown in Table 14.  
Considering the relatively small size of the dataset, we conduct 5-fold cross-validation (CV) at the utterance level with stratification with regard to utterance labels, so that (1) the class distribution in each fold is close to being identical, and (2) each time, 4 folds are used as training and validation data and 1 fold is used as test data. The training to validation data ratio is 9:1, and we select the best-performing checkpoint (for CNN and BERT models) based on the performance on the validation set, so that we can test the checkpoint on the test set. We use Macro F1 as the validation and test metric, since it is commonly used for classification tasks and robust to class imbalance. For Prior and Uniform, whose outputs are random, we run the models 1000 times on the test data and calculate the average performance. Therefore, each of the 6 models listed above eventually has 5 performance values from 5-fold CV, and we take the mean as the final performance value of the model.