# Creation, Analysis and Evaluation of AnnoMI, a Dataset of Expert-Annotated Counselling Dialogues +
## 3 Creating AnnoMI
### Expert Annotators and Workload Assignment  
As MI annotation requires specialised knowledge, we rely on experienced MI practitioners to annotate the transcripts. Specifically, 10 therapists found through the Motivational Interviewing Network of Trainers ([[https://motivationalinterviewing.org/trainer-listing](https://motivationalinterviewing.org/trainer-listing)]([https://motivationalinterviewing.org/trainer-listing](https://motivationalinterviewing.org/trainer-listing)), accessed on 14 February 2023), an international organisation of MI trainers and a widely recognised authority in MI, were recruited for the task. All the annotators had a high proficiency in English and prior experience in practising and coding MI. Informed consent was collected from all the annotators. \begin{table}
\begin{tabular}{p{227.6 pt}} \hline \hline \multicolumn{2}{c}{**High-Quality MI**} \\ \hline \hline
**T**: Um, I did wanna talk to you though. I’m a little bit concerned looking through his chart of how many ear infections he’s had recently. And I-I noticed that you had checked the box that someone’s smoking in the home. So I was wondering if you can tell me a little more about that. \\
**C**: Well, um, It’s just me and him and I do smoke. Um, I try really hard not to smoke around him, but I-I’ve been smoking for 10 years except when I was pregnant with him. But it– everything is so stressful being a single mom and-and my having a full-time job. And so it’s just—that’s why I started smoking again. \\
**T**: You have a lot of things going on and smoking’s kind of a way to relax and destress. \\
**C**: Yeah. Some people have a glass of wine. I have a cigarette. \\
**T**: Sure. And it sounds like you’re trying not to smoke around him. Why did you make that decision? \\ \hline \hline \multicolumn{2}{c}{**Low-Quality MI**} \\ \hline \hline
**T**: Well, now’s the time to quit. It’s really gotten to the point where you can’t keep smoking. Not only for him, like I said, but also for you. You’re putting yourself at risk for lung cancer, for emphysema, for oral cancers, for heart disease, for all kinds of things- \\
**C**: I know, I know. I’ve heard– People have told me before, I’ve heard all that. I just don’t know how to do it. How am I supposed to quit? It’s-it’s so hard. \\
**T**: Well, there’s all kinds of things you can use now. It’s not as hard as it used to be. You can use nicotine replacement. There’s patches, there’s lozenges, there’s gum, there’s the inhaler, there’s nasal spray. We can talk about medications. You can try Chantix, you can try Zyban, there’s quit smoking groups you can go to, there’s hotlines you can call. \\
**C**: I just don’t have time for any of that. \\ \hline \hline \end{tabular}
\end{table} 
Table 1: High- and low-quality MI conversation snippets, where the goal is smoking cessation/reduction. _T_: therapist; _C_: client. Overall, each expert annotated 19 to 20 transcripts, with total lengths around 144 min in terms of the total duration of the original 19 to 20 videos. To facilitate computation of inter-annotator agreement (IAA), we selected 7 common transcripts to be annotated by all experts, based on 3 criteria: (1) they should add up to approximately 1/3 (45 min) of the workload of each annotator; (2) they should cover diverse topics (6 out of the 7 transcripts have distinct topics); (3) they should cover both high- and low-quality demonstrations (5 showing high-quality MI and 2 showing low-quality). We tried various combinations of transcripts before we found one combination that satisfied the criteria above. During the annotation process, no expert was aware that a part of their workload would be used to compute the IAA. Each of the 126 (133-7) non-IAA transcripts was annotated by one expert due to our budget limits.