# Creation, Analysis and Evaluation of AnnoMI, a Dataset of Expert-Annotated Counselling Dialogues +
## 1 Introduction  
Patient health can be significantly improved by changes in behaviour, such as reducing alcohol consumption [1]. Counsellors, however, may have difficulty in convincing patients to adopt such changes. Thus, motivational interviewing (MI) [2] has been developed as an effective counselling approach that evokes motivation for change from the client themselves (A client in counselling is not necessarily a patient, so the word \"client\" is used in this study). Correspondingly, coding systems such as MISC (Motivational Interviewing Skill Code) [3] and MITI (Motivational Interviewing Treatment Integrity) [4] are commonly used to identify MI codes and aspects related to the therapist and client. Recent years have seen significant interest in the research of linguistic and statistical MI analysis. The first computational model for identifying reflection, a key skill in MI, was introduced by [PERSON] et al. [5]. More broadly, the modelling of MI-related aspects such as codes and therapist empathy has been approached with methods based on classical machine learning [6, 7, 8] (e.g., support vector machines) and deep learning [9, 10, 11, 12] (e.g., recurrent neural networks). In terms of data resources, [PERSON] et al. [13] recently published acorpus of MI conversations automatically transcribed from video-sharing websites, where some dialogues showcase high-quality MI and the others illustrate low-quality MI. Despite the progress, natural language processing (NLP) for MI is still an extremely low-resource domain, owing to privacy-related restrictions. As research in this field has been conducted primarily on private/undisclosed annotated MI dialogues, it has been challenging to replicate and further develop prior work. Previously, to the best of our knowledge, the only publicly available dataset of MI conversations was created by [PERSON] et al. [PERSON] _et al._ (2018) through the automatic captioning of YouTube/Vimeo videos. However, the transcript quality is compromised by the considerable transcription errors from automatic captioning, which can make the transcripts difficult to understand. [PERSON] et al. [PERSON] _et al._ (2018) also analysed two MI codes--reflection and question--based on the dataset annotations from trained students, but these annotations are unavailable at the time of writing. To alleviate the scarcity of resources for NLP-for-MI research, we previously introduced AnnoMI [PERSON] _et al._ (2018), a dataset of 133 MI-adherent and non-adherent therapy conversations that are (a) professionally transcribed from YouTube/Vimeo videos demonstrating MI, and (b) annotated for key MI aspects by experienced MI practitioners. We also obtained explicit permission from the video owners for creating, releasing, and analysing the dataset. We note that \"MI-adherent\" and \"MI non-adherent\" are synonyms for \"high-quality\" and \"low-quality\" respectively, which describe therapy quality rather than video/transcription quality. In this work, we expand [PERSON] _et al._ (2018) significantly as follows:  
1. We release the full version of AnnoMI, which has several fine-grained additional attributes. We also elaborate on the details of data collection and processing, including the results of a post-annotation survey for the annotators, which suggest that the dataset reflects real-world high- and low-quality MI even though its dialogues are from demonstration videos. 2. We present detailed, visualised statistical analyses of the expanded dataset to examine its patterns and properties. 3. We establish two AnnoMI-based utterance-level classification tasks with potential for real-world applications: therapist behaviour prediction and client talk type prediction. We also experiment with various machine learning models as baselines for these tasks to facilitate comparison with future methods. 4. We explore the performance of these models on different topics, as well as their generalisability to new topics.