<figure><img src="image_1.png" /></figure>

HAL
open science

# EMMI-Empathic Multimodal Motivational Interviews: automatic multimodal features extraction and patient type definition

Lucie Galland, Catherine Pelachaud, Florian Pecune

► To cite this version:

Lucie Galland, Catherine Pelachaud, Florian Pecune. EMMI-Empathic Multimodal Motivational Interviews: automatic multimodal features extraction and patient type definition. Language Resources and Evaluation, 2025, 10.1007/s10579-025-09856-x. http://hal-05365230

HAL Id: hal-05365230

https://hal.science/hal-05365230v1

Submitted on 14 Nov 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
EMMI - Empathic Multimodal Motivational
Interviews: Automatic Multimodal Features
Extraction And Patient Type Definition

Lucie Galland¹*, Catherine Pelachaud² and Florian Pecune³

¹*ISIR, Sorbonne university, Paris, France.

²CNRS - ISIR, Sorbonne university, Paris, France.

³CNRS - SANPSY, Univ. Bordeaux, Bordeaux, France.

*Corresponding author(s). E-mail(s): lucie.galland@isir.upmc.fr;

Contributing authors: catherine.pelachaud@isir.upmc.fr;

florian.pecune@u-bordeaux.fr;

Abstract

The increasing prevalence of mental health issues has created a significant gap between the demand for mental health services and available resources, resulting in prolonged wait times for therapy. Virtual agents capable of conducting Motivational Interviews (MI) offer a promising solution by providing immediate support and interventions during the waiting period. Our aim is to build such an agent, a task which first requires analyzing human-human data so we can build a computational model to drive the agent's verbal and nonverbal behaviors. This study presents the Empathic Multimodal Motivational Interviews (EMMI) features set, composed of multimodal features extracted from two existing MI corpora, AnnoMI and the Motivational Interviewing Dataset (MID). Our analyses focus on therapists' and patients' verbal and nonverbal behaviors to understand their interactions better.

We identify three distinct patient types, "Open to change," "Resistant to change," and "Receptive", and examine how therapists adapt their behaviors to their clients. Our findings reveal that therapists adjust their verbal and non-verbal behavior based on patients' types. This study highlights the importance of considering verbal and nonverbal cues when developing virtual agents for MI, emphasizing the need for these agents to adapt dynamically to different patients' types and behaviors. Our insights provide a foundation for creating adaptive virtual MI interviewers to personalize interventions, thus enhancing their effectiveness and supporting mental health care delivery.
# 1 Introduction

The frequency of mental health issues has notably increased, resulting in a substantial gap between the demand for mental health services and the available resources (Cameron et al, 2017). Consequently, patients often face prolonged waiting times before beginning therapy (Cameron et al, 2017; Denecke et al, 2020). To address this issue, virtual agents capable of replicating Motivational Interviews (MI) for patients awaiting appointments offer a promising solution. These virtual agents can potentially alleviate the waiting period by providing immediate support and interventions, particularly in therapeutic contexts involving multiple sessions (Fiske et al, 2019). The effectiveness of such agents or chatbots has been proven in several studies (Bickmore et al, 2009, 2018; Fitzpatrick et al, 2017; Ring et al, 2016).

MI is a collaborative therapeutic approach that encourages behavioral change. During Motivational Interview sessions, therapists use various strategies to help patients articulate their motivation for change (Miller and Moyers, 2017). These strategies encompass verbal elements (e.g., reflections, questions) and nonverbal elements (e.g., smiles, head and body positions) (Younsi et al, 2024). A virtual agent employing both verbal and nonverbal strategies to conduct motivational interviews could effectively rekindle the user's motivation when needed (Mercado et al, 2023; Bickmore et al, 2009, 2018).

Although previous MI agents tend to have non adapted behaviors, MI is by definition a cooperative and adaptive type of communication. Therefore, the development of an agent able to adapt its dialogue to the situation could be significant for the efficacy of the therapy (Creber et al, 2016). Creating such an agent necessitates examining human-human data to gain insights into therapist's behaviors during these interviews and detect variations in patient's behaviors. However, obtaining this data is challenging due to the sensitive nature of the topics discussed. Investigating the multimodal conduct of therapists would provide invaluable insights for developing the agent, as psychological theory emphasizes the significance of nonverbal behavior during these interviews (Gillam and Yusuf, 2019). Another crucial aspect highlighted by psychological theory is the role of empathic and social conduct. Patients are more likely to be honest and share their symptoms and concerns with an empathic physician, leading to better therapy outcomes (Jani et al, 2012). Clinical empathy involves perceiving the patient's world experience and understanding their current feelings.

In this paper, we present an analysis of interactional behaviors based on multimodal features extracted from two existing MI corpora, namely AnnoMI (Wu et al, 2023) and the Motivational Interviewing Dataset (MID) (Pérez-Rosas et al, 2019). These features aim to scrutinize the empathic conduct of therapists during MI sessions and the behaviors and responses of patients to such empathic interactions. Collectively, these features form the Empathic Multimodal Motivational Interviews (EMMI) features set.
This paper makes the following contributions: First, we extract multimodal features from publicly available MI corpora, creating the EMMI features set. This includes the development of a classifier for dialogue acts tailored for therapy settings. Second, we analyze patient's behaviors, identifying three distinct types of patients. Finally, we examine therapists' multimodal behaviors according to the type of patient they interact with.

We present the background of the study and existing corpora in Section 2. Then, EMMI, our set of extracted features, is introduced. Section 5 describes our annotation process and the types of MI patients are introduced in Section 5.10. In Section 6, we analyze our features to highlight differences in the behaviors of patients and therapists depending on their type, and whether the observed behaviors occur during the first or the second half of the conversation. Finally, in Section 7, we interpret these differences and discuss the impact of types on both the patient's and therapist's behavior.

# 2 Background and Existing Corpora

In this section, we present background on motivational interviewing as well as different preexisting corpora.

## 2.1 Motivational interviewing

Motivational Interviewing is an approach to therapy that emphasizes collaboration and encourages behavioral change. During MI, therapists employ strategies to guide patients toward expressing motivation to change (Miller and Rollnick, 2012). One primary strategy is the use of reflections, where therapists convey their understanding of what the patient is saying or feeling without judging, interpreting, or advising, leading the client to reflect on their situation and actions.

Patient behaviors (including speech, prosody and non verbal behaviors) during these dialogues typically fall into three categories as defined by the Motivational Interviewing Skill Code (MISC) (Miller et al, 2003):

* **Change talk (CT)**: Actions toward behavior change, the client shows behavior indicating that they are willing to change their unhealthy behavior.

* **Sustain talk (ST)**: Actions away from behavior change, the client shows behavior indicating that they are unwilling to change their unhealthy behavior.

* **Follow/Neutral (F/N)**: Unrelated to the target behavior.

This classification of client language is crucial as it predicts therapy outcomes. Indeed, (Magill et al, 2014) revealed that **sustain talk** was associated with poorer treatment results. Furthermore, (Magill et al, 2018) demonstrated that change talk was linked to risk behavior reduction during follow-up assessments. Beyond the goal-oriented aspect of the dialogue focused on reflections and change language, MI interviewers also employ social strategies to build relationships with patients (Miller and Moyers, 2017). These strategies manifest themselves in verbal behaviors through empathic reactions and encouragement (Angus and Kagan, 2009), in nonverbal behaviors such as smiles (Younsi et al, 2024) as well as verbal and non-verbal alignment (Lord et al,
2015). Building a strong relationship with the patient is crucial as it fosters a sense of confidence, encouraging the patient to share more information (Moyers et al, 2005).

## 2.2 Existing corpora

Numerous studies focus on exploring Motivational Interviewing at the verbal level (Wu et al, 2022; Pérez-Rosas et al, 2016; Tavabi et al, 2020). However, a significant challenge arises from the unavailability of relevant data due to the sensitive nature of the topics discussed within MI conversations. Most human-human MI corpora cannot be publicly disseminated and are privately owned. For instance, research by (Pérez-Rosas et al, 2016, 2017b) analyzes therapist behavior using phone conversation transcripts of actual therapy sessions, and (Pérez-Rosas et al, 2017a) collected a private corpus of audio-recorded therapy session interactions to study therapist empathy, but these datasets cannot be shared. (Tavabi et al, 2020) focuses on patient behavior classification using transcribed audio data, with only publicly available text transcripts.

The aforementioned corpora are based on audio recordings, making the study of nonverbal behaviors impossible. Although some video-based corpora have emerged, such as the Counseling and Therapy Video Library, they are privately owned and often require substantial acquisition costs. Only a few of the existing corpora (not publicly available) could be used to study facial expressions and body language; for example, (Nakano et al, 2022) assembles a video corpus for detecting change talk, but it is not publicly available.

However, recent efforts have been made to create publicly available datasets. (Pérez-Rosas et al, 2019) introduces ANNOMI, a corpus of MI videos scraped from the web, though automatic YouTube captioning leads to transcription errors. Additionally, (Wu et al, 2023) provides a publicly accessible MI video corpus transcribed and annotated by experts. ANNOMI comprises high-quality interactions designed to teach good practices in MI and low-quality interactions designed to show mistakes to avoid during an MI intervention. These studies primarily focus on text analysis, particularly the behavior of therapists of varying quality (Wu et al, 2022; Pérez-Rosas et al, 2019),.

However, none of these publicly available resources focus on video and audio modalities. They do not explore how patients' behavior varies during a session and how therapists adapt their strategies during the interaction. This paper addresses this gap by comprehensively developing and analyzing therapists' and clients' multimodal behavior.

## 3 Research Questions

The study of multimodal interactions in therapy aims to reach a comprehensive understanding of therapist and patient behavior, which can be used to develop a multimodal virtual agent supporting therapy. Currently, most datasets in this field primarily focus on the verbal aspect, only considering limited information from nonverbal behavior. Existing studies (Wu et al, 2022; Pérez-Rosas et al, 2019) often concentrate on comparing high- and low-quality interactions, neglecting a deeper analysis of social behaviors. Our research analyzes therapy session videos, considering the patients' and
therapists' progression on verbal and nonverbal behaviors and their alignment. Our primary objective is to examine the evolution of the therapist's and patient's verbal and nonverbal behaviors throughout the interaction. Furthermore, this study aims to uncover how patients respond to different therapeutic strategies and identify cues that are correlated with patients' behavior expressing change talk, leading to categorizing patients into various "types" requiring tailored therapeutic approaches. Lastly, we strive to determine the reciprocal influence of behaviors between the two speakers during the therapeutic process.

We address the following research questions:

**RQ1:** How do the verbal and nonverbal behaviors of therapists and patients evolve throughout the interaction?

**RQ2:** Can distinct types of patients be identified based on changes in talk patterns related to their willingness to change behavior, and to what extent do these types influence the behavior of both patients and therapists during therapy?

**RQ3:** How do therapists adapt their verbal and nonverbal behaviors in response to patients' expressions of change, neutral, or sustain talk regarding their unhealthy behaviors?

By investigating these research questions, we aim to gain insights into multimodal interaction during therapy and contribute to the development of effective and personalized virtual therapeutic interventions.

# 4 EMMI

We gathered the videos of AnnoMI ((Wu et al, 2023) and MID (Pérez-Rosas et al, 2019) into a single set. We got 285 videos with 47 turns on average, representing 21 hours and 22 minutes of interactions. EMMI corresponds to the set of behaviors we extracted from these videos. It is a set of multimodal features that are described below; all the extracted features are available on OSF<fnref n="1" />. These videos consist of simulated motivational interview conversations in which actors portray the roles of patients and therapists. However, these corpora were initially designed for natural language analysis. Only the speech was transcribed. There is no annotation of multimodal features. Our study aims to fill this gap by extracting and analyzing the multimodal elements within these videos, enabling a more comprehensive understanding of therapeutic interactions. These corpora focus on the task-oriented aspects of the conversation, with features that capture the quality of MI provided. We extract additional features to study the social aspect of motivational interviewing.

## 4.1 AnnoMI

AnnoMI is a publicly available dataset of 132 expert-annotated MI videos. 3 videos could not be used because they were no longer accessible, and 9 were removed because of poor video quality. The selected part of AnnoMI represents 13 hours and 25 minutes of videos. The videos are transcribed and classified between high- and low-quality MI. Transcripts are segmented into utterances, with each new utterance beginning upon an intervention by the other speaker. As a result, even brief interruptions such as

<footnote n="1">¹https://osf.io/wz4vr/</footnote>
backchannels divide a single speaker turn into multiple utterances. Each utterance is annotated using MISC based on the primary therapist's verbal behavior (question, reflection, other) and the patient's talk type (neutral, change, sustain). In the following, we focus on the high-quality of MI part of the corpus as we aim to study high-quality therapists' behavior to later develop a high-quality virtual therapist.

## 4.2 Motivational Interviewing Dataset (MID)

MID is composed of 247 publicly available simulated MI conversations. 39 of these are removed because some of the videos are already part of the AnnoMI dataset, 37 are missing, and 6 are of poor video quality. The selected part of the corpus represents 7 hours and 56 minutes. The videos are transcribed, no other annotation is provided.

# 5 Multimodal features

In the following sections, we analyse the different features extracted from the therapist and patient multimodal behaviors. Using automatic tools we extracted the dialogue acts, body movements and facial expressions. From these features, we then conducted several analyses. We examine task-related behaviors, encompassing the dialogue acts used during the conversation that reflect the classical steps of MI (Beauvais, 2019). Additionally, we explore the concept of behavior expressivity, gauged through indicators like gesture arm openness and voice loudness, which offer insights into the confidence levels of the speakers and are also a sign of empathy (Decker et al, 2014). This dimension helps us understand how self-assured the therapist and patient feel during the interaction. Our investigation also encompasses social-related behaviors, such as smiles and social dialogue acts, which provide valuable insights into the evolving dynamics and rapport between the therapist and the patient (Jani et al, 2012). Finally, we look at the alignment between the therapist and the patient as it is one of the indicators of therapists' empathy that has been found to be correlated with treatment outcome (Lord et al, 2015). We focus on verbal alignment (how much verbal expressions of the speaker are reused) and behavior expressivity alignment (how similar the behavior expressivity of the two speakers are). Since the faces of both speakers are not always fully visible in the videos, and often from the side when both are visible, we choose not to analyze facial expression alignment. The extracted features are accessible on OSF<fnref n="2" />.

## 5.1 Videos pre-processing

In preparation for automated feature extraction, the videos from the MID were transcribed anew, because of the bad quality of the original transcriptions, using the transcription service Descript. To facilitate consistent analysis and comparison of simultaneous nonverbal behavior, each original video, originally recorded with varying camera perspectives (e.g., patient-only, therapist-only, or both side by side), is reproduced to generate three distinct standardized versions: one video showing only the patient, one showing only the therapist, and one showing both participants side by side.

<footnote n="2">²https://osf.io/wz4vr/</footnote>
<figure><img src="image_3.png" /><figcaption>(a) Original video frame showing both patient and therapist side by side.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>(b) Edited video frame showing only the patient, with the therapist obscured using a black rectangle.
(c) Edited video frame showing only the therapist, with the patient obscured using a black rectangle.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>(d) Original video frame showing only the patient.</figcaption></figure>

<caption>(e) Original video frame showing only the therapist.</caption>

**Fig. 1:** Standardization of video views for multimodal behavior analysis. From the original dual-view video format showing both patient and therapist, two additional standardized views are created using manual masking. This process ensures consistent visibility conditions for analyzing non-verbal behavior in MI sessions.
<figure><img src="image_6.png" /><figcaption>**Fig. 2:** Example of our separation of AnnoMI utterance into speaking turns with semantic meaning. Here, two utterances were separated, in the AnnoMI corpus, by a backchannel. We regrouped them into one speaking turn.</figcaption></figure>

This is achieved by manually editing each video frame by frame, adding black rectangles where necessary to obscure the non-target speaker, ensuring a clean and consistent view for each perspective(see Fig.1). By deriving three distinct video versions from each original: patient-only, therapist-only, and dual-view, we enable efficient identification and comparison of each individual's visible non-verbal behavior. Because the original recordings employed different camera angles, non-verbal behaviors were often not visible for one of the speakers during portions of the interaction. This limitation needs to be accounted for in subsequent analyses. To address this, we extracted two additional features: (1) a frame-level annotation indicating who is visible in the video (patient, therapist, or both), and (2) an annotation of speaker identity, specifying who holds the speaking turn at each moment (i.e., the therapist or the patient).”

## 5.2 Separation into speaking turns

To analyze the interaction, the corpus is separated manually into speaking turns. A speaking turn is composed of a sequence of utterances performed by the same speaker (therapist or patient). During a speaking turn, the listener can produce a backchannel that does not interrupt the turn. To perform this separation, we extracted backchannels from the transcript and regrouped interrupted utterances into complete speaking turns. An example of this reorganization is depicted in Figure 2. For each speaking turn, the features set<fnref n="3" /> includes the number of listener backchannels, along with their timing and corresponding transcripts.

<footnote n="3">³The extracted features are accessible on OSF: https://osf.io/wz4vr/</footnote>
<table><caption>**Table 1:** Classification scores of patients dialogue act. The 95% confidence intervals are computed using the bootstrap method and a 1000 runs (Efron and Tibshirani, 1994)</caption><thead><tr><th></th><th>Definition</th><th>Macro F1 score</th><th>Accuracy</th></tr></thead><tbody><tr><td>Changing unhealthy behavior</td><td>The patient explicitly expresses their willingness to change</td><td>0.80[0.76;0.84]</td><td>0.89[0.87;0.91]</td></tr><tr><td>Sustaining unhealthy behavior</td><td>The patient explicitly expresses their unwillingness to change</td><td>0.61[0.55;0.67]</td><td>0.89[0.87;0.91]</td></tr><tr><td>Sharing negative feeling or emotion</td><td>The patient shares a negative feeling or vision of the world</td><td>0.63[0.59;0.68]</td><td>0.80[0.78;0.83]</td></tr><tr><td>Sharing positive feeling or emotion</td><td>The patient shares a positive feeling or vision of the world</td><td>0.65[0.54;0.77]</td><td>0.98[0.96;0.99]</td></tr><tr><td>Realization or Understanding</td><td>The patient realized or understood something about their problem</td><td>0.63[0.55;0.71]</td><td>0.94[0.93;0.96]</td></tr><tr><td>Sharing personal information</td><td>The patient shares factual personal information about their situation or background</td><td>0.69[0.65;0.72]</td><td>0.70[0.66;0.73]</td></tr><tr><td>Greeting or Closing</td><td>The patient opens or closes the conversation</td><td>0.76[0.65;0.85]</td><td>0.98[0.96;0.99]</td></tr><tr><td>Backchannel</td><td>The patient acknowledges that they heard the last therapist's statement</td><td>0.65[0.59;0.72]</td><td>0.90[0.89;0.92]</td></tr><tr><td>Asking for medical information</td><td>The patient asks for medical information</td><td>0.74[0.57;0.87]</td><td>0.99[0.98;0.99]</td></tr><tr><td colspan="2">Macro average</td><td>0.69[0.65;0.72]</td><td>0.90[0.89;0.90]</td></tr></tbody></table>

## 5.3 Dialogue act

In this section, we introduce an automatic classifier for MI dialogue acts. This classifier is validated against manual annotations of a subpart of the corpus and then used to automatically classify the rest of the corpus. Dialogue acts were annotated using a schema derived from (Chiu et al, 2024). Patients' speaking turns were automatically classified into one or multiple of the dialogue acts defined in Table 1. Therapists' speaking turns were classified into either one task-oriented dialogue act or one socially oriented dialogue act described in Table 2. The speaking turns were classified using Mistral instruct (Jiang et al, 2023) and few shot learning techniques. Mistral instruct is an open-source, deep learning based large language model usable with an 8GB GPU. The prompt is derived from (Chiu et al, 2024) and includes definitions of the possible dialogue acts as well as examples of speaker turns. The prompts and associated code are provided on Github<fnref n="4" />. The automatically extracted features were tested on a subpart of the HOPE dataset (Malhotra et al, 2022). The HOPE dataset comprises therapy videos with good-quality transcripts overlapping with AnnoMI and MID. The resulting automatic classifier will, therefore, be transferable to EMMI. We used HOPE to test the classifier and validate it. 500 speaking turns of the patient and 500 of the therapist have been annotated twice, with one week delay, by one of the paper's authors, with a Cohen kappa alpha agreement of 0.65 for the patient and 0.7 for the therapist. These annotated speaker turns were used to validate the automatic extraction with an average macro-F1 score of 0.69 for the patient and the therapist (see F1 values for each dialogue act in 1 for the patient and in 2 for the therapist). The prompts and the annotations are available on Github<fnref n="4" />.

<footnote n="4">⁴https://github.com/l-Galland/MI-DA-classification</footnote>
<table><caption>**Table 2:** Classification scores of therapists' dialogue act. The 95% confidence intervals are computed using the bootstrap method and a 1000 runs (Efron and Tibshirani, 1994)</caption><thead><tr><th></th><th>Definition</th><th>Macro F1 score</th><th>Accuracy</th></tr></thead><tbody><tr><td colspan="4"><strong>Task oriented Dialogue acts</strong></td></tr><tr><td>Ask for consent or validation</td><td>The therapist checks that their last statement was correct or that the patient consented to move forward</td><td>0.67[0.55;0.78]</td><td>0.97[0.96;0.98]</td></tr><tr><td>Medical Education and Guidance</td><td>The therapist provides the patient with medical or therapeutic facts</td><td>0.83[0.74;0.89]</td><td>0.97[0.96;0.98]</td></tr><tr><td>Planning with the patient</td><td>The therapist builds a plan with the patient to modify their unhealthy behavior/thoughts</td><td>0.73[0.67;0.79]</td><td>0.92[0.90;0.94]</td></tr><tr><td>Give Solution</td><td>The therapist provides the patient with solutions to solve their problem</td><td>0.66[0.59;0.73]</td><td>0.94[0.92;0.95]</td></tr><tr><td>Ask about current emotions</td><td>The therapist ask the patient what they are feeling during the therapy session</td><td>0.64[0.60;0.69]</td><td>0.88[0.85;0.90]</td></tr><tr><td>Invite to shift outlook</td><td>The therapist asks the patient to imagine their reaction to a future event or to change their perspectives on a past even</td><td>0.57[0.51;0.63]</td><td>0.91[0.88;0.93]</td></tr><tr><td>Ask for Information</td><td>The therapist asks the patient for factual information about their background or situation</td><td>0.78[0.75;0.81]</td><td>0.80[0.77;0.83]</td></tr><tr><td>Reflection</td><td>The therapist summarizes or reformulates the patient statement without judgment</td><td>0.73[0.69;0.76]</td><td>0.77[0.74;0.80]</td></tr><tr><td colspan="4"><strong>Socially oriented Dialogue acts</strong></td></tr><tr><td>Empathic reaction</td><td>The therapist expresses empathy to the patient</td><td>0.62[0.53;0.71]</td><td>0.96[0.94;0.97]</td></tr><tr><td>Acknowledge progress and encourage</td><td>The therapist praises the patient for their achievements or encourages them</td><td>0.62[0.53;0.71]</td><td>0.96[0.94;0.97]</td></tr><tr><td>Backchannel</td><td>The therapist acknowledges that they heard the patient's statement</td><td>0.65[0.56;0.74]</td><td>0.96[0.95;0.97]</td></tr><tr><td>Greeting or Closing</td><td>The therapist opens or closes the conversation</td><td>0.85[0.77;0.93]</td><td>0.98[0.96;0.95]</td></tr><tr><td>Experience Normalization and Reassurance</td><td>The therapist normalizes the patient's experience and reassure them</td><td>0.59[0.49;0.68]</td><td>0.97[0.95;0.98]</td></tr><tr><td colspan="2">Macro average</td><td>0.69[0.67;0.71]</td><td>0.92[0.91;0.93]</td></tr></tbody></table>

## 5.4 Verbal alignment

During an interaction where participants are highly engaged, an interlocutor may tend to reuse vocabulary introduced by the other speakers (Campano et al, 2014). Iteratively, the speakers create a common vocabulary database. We call this verbal alignment. We extract verbal alignment with the package Dialign (Dubuisson Duplessis et al, 2021). This package computes the number of verbal expressions initiated by one speaker and reused by the other. It describes the verbal alignment between two speakers during a dialog. We were interested in the number of expressions reused per turn.

## 5.5 Loudness

During a dialog, speakers can use different levels of voice intensity and speak more or less loudly. The volume of the voice or loudness can be a sign of emphasis on a word or be a sign of a dominant attitude (Decker et al, 2014). We extract the loudness of the voice using OpenSmile (Eyben et al, 2010).
<figure><img src="image_7.png" /><figcaption>Fig. 3: definition of *L* and *H* given the keypoints extracted by OpenPose</figcaption></figure>

## 5.6 Facial Action Units

The Facial Action Units (AUs) introduced by (Friesen and Ekman, 1978) are a set of visible facial muscle movements. AUs of the therapist and the patient were automatically collected using OpenFace for the frames where the information is available. (Baltrusaitis et al, 2018). OpenFace does not output any action units for certain frames because the model failed to detect any faces; the output data is, therefore, interpolated for missing values. A median filter of size three is used to smooth the features and remove classification errors.

## 5.7 Smile

Smiles were detected using the HMAD project (Rauzy and Goujon, 2018) that annotates smiles on the five levels of the Smiling Intensity Scale based on the features extracted with OpenFace.

## 5.8 Arms Openness

The openness of the arms is a sign of open or closed posture that can be an observable consequence of different social attitudes, such as domination or confidence (Decker et al, 2014). To extract the arms' openness, we first extract the body joints' position and orientation using OpenPose (Cao et al, 2017). The arms' openness is then defined as the width of the bounding box around the speaker for a given time frame. It is computed by dividing the distance between the two wrists by the height of the bust in the current frame $O = \frac{L}{H}$ (see Fig 3).

We normalize by the height of the bust to take into account different camera positions and people's heights.
## 5.9 Change talk

As already presented in section 2, patient talk can be classified into three distinct classifications: Change talk, Sustain Talk, and Neutral Talk. The MISC classification scheme carries significance due to its demonstrated correlation with therapy outcomes (Magill et al, 2014, 2018). The talk type has already been annotated within AnnoMI. Previous work trained a multimodal classifier using this annotated data, yielding an F1 score of 0.8 (Galland et al, 2024). This classifier is employed to annotate the MID in terms of **change talk**<fnref n="5" />.

## 5.10 Patient clustering into types

As previously discussed (see Section 5.9), the EMMI features set includes annotations for the level of change talk, categorized into three classes: neutral, sustain, and change. This annotation is transformed into a numerical score of change talk where **sustain** = -1, **neutral** = 0, and **change** = 1. We apply the KMeans algorithm with a DTW metric (Sakoe and Chiba, 1978) to compute clusters of patients. The DTW metric accounts for similar patterns in change talk behavior even when the timing or length of the pattern is not precisely identical. The Elbow method (Thorndike, 1953) shows that the optimal number of clusters is 3 with a silhouette score of 0.17. The visualization of the clusters of patients, as well as the comparison between the proportion of change and sustain talk in the first half of the interaction and the second half of the interaction (see Fig. 4) shows three types that can be concretely interpreted.

Patient type (Kruskal-Wallis H test: $H_{change}(2) = 18.1, p_{change} = .00012e$, $H_{sustain}(2) = 20.5, p_{sustain} = .000035$) and Time (Kruskal-Wallis H test: $H_{change}(1) = 40.7, p_{change} < .000001$, $H_{sustain}(1) = 12.17, p_{sustain} = .000051$) have a significant main effect on both **change** and **sustain talk** proportion for at least two groups. The following Mann-Whitney U post hoc test results can be seen in Table 3 and Figure 4. Since (for Substance Abuse Treatment et al, 2019) states that "Patient stuck in ambivalence engages in a lot of **sustain talk**, whereas patients who are more open to change engage in more **change talk** with stronger statements supporting change, we can then interpret our clusters as the following:

* When the patient begins the conversation with significantly less **sustain talk** than other patients ($p < .05$) and significantly more **change talk** than other patients ($p < .1$), we define them as **"Open to change"** (cells in green in Table 3)

* When the patient displays significantly less **change talk** throughout the whole conversation than other patients ($p < .05$), we define them as **"Resistant to change"** (blue cells in Table 3)

* When the patient starts with a comparable number of **sustain talk** as "Resistant to change" patients but more **change talk** ($p < .05$) and the patient displays significantly more **change talk** than "Resistant to change" patient during the second half of the conversation ($p < .05$), we define them as **"Receptive"** (cells in red in Table 3)

<footnote n="5">⁵This classification into **change**/**sustain**/**neutral** is different from the one carried out in Section 5.3 as it considers all modalities and not only the text. This classification also takes into account subtext that is not taken into account by the dialogue act classifier and is therefore complementary.</footnote>
<table><thead><tr><th></th><th>p change</th><th>p sustain</th></tr></thead><tbody><tr><td>1st half</td><td>&lt;.0000001 (****)</td><td>.000039 (****)</td></tr></tbody></table>

<table><tr><td>Receptive</td><td>Open to change</td><td>.95</td><td>.00000052 (****)</td></tr><tr><td>Open to change</td><td>Resistant to change</td><td>.0038 (**)</td><td>.0000026 (****)</td></tr><tr><td>Receptive</td><td>Resistant to change</td><td>.02 (*)</td><td>.55</td></tr></table>

<table><tr><td rowspan="3">1st half of the interaction</td><td>Receptive</td><td>Open to change</td><td>.097 (.)</td><td>.00000012 (****)</td></tr><tr><td>Open to change</td><td>Resistant to change</td><td>.000012 (****)</td><td>.012 (*)</td></tr><tr><td>Receptive</td><td>Resistant to change</td><td>.026 (*)</td><td>.19</td></tr></table>

<table><caption>**Table 3:** Post hoc test on patient talk type depending on patients type and time. p<.1, * p<.05, ** p<.01, ** p<.001, *** p<.0001, **** p<.00001</caption><tr><td rowspan="3">2nd half of the interaction</td><td>Receptive</td><td>Open to change</td><td>.26</td><td>.16</td></tr><tr><td>Open to change</td><td>Resistant to change</td><td>.11</td><td>.013 (*)</td></tr><tr><td>Receptive</td><td>Resistant to change</td><td>.043 (*)</td><td>.61</td></tr></table>

<figure><img src="image_8.png" /><figcaption>**Fig. 4:** Evolution of **change** and **sustain talk** in time according to patient type. p<0.1, * p<0.05, ** p<0.01, ** p<0.001, *** p<0.0001, **** p<0.00001</figcaption></figure>

## 6 Multimodal behavior analysis

In this section, we investigate our three research questions through the targeted behaviors described in Section 5 divided into 3 categories: task behavior, expressivity, social behavior, and alignment. Section 7 discusses the reported statistical results. We analyze the differences in target behaviors between the first and second half of the conversation to study the evolution during the dialogue (RQ1). To this intent, we separated the interaction into two parts, as the small number of videos does not allow for more fine-grained studies. We also study the differences between different types of
patient talk (RQ2) and talk types (RQ3). We use non-parametric tests whenever the normality requirements for ANOVA were not met. All p-values were corrected using the Bonferroni method. For clarity purposes, we do not report every test value in the following section.

## 6.1 Task related behavior

In this section, we analyze statistical differences in behaviors related to the task, respectively therapist and patient dialogue acts, according to different patient types and timing in the dialog.

### 6.1.1 Therapist dialogue acts

The therapist plans more with the patient during the second half than during the first half of the conversation (T-test: *p* = 0.02). There is also a difference in the use of planning according to at least two groups of patient types (Kruskal-Wallis H test: *p* = 0.03). Therapists plan significantly more with "Receptive" than with "Resistant to change" (Mann-Whitney U test: *p* = 0.04). The therapist asks for more information during the first half of the conversation (T-test: *p* = 0.003), but there were no significant differences between patients' talk types (Kruskal-Wallis H test: *p* = 0.84). There were no significant differences between the first and the second half of the conversation in the use of invitation to shift outlook (T-test: *p* = 0.12). However, there is a difference between at least two groups of patients for the invitation to change outlook (Kruskal-Wallis H test: *p* = 0.02). The therapist invites significantly more to shift outlook when interacting with a "Receptive" patient than when interacting with a "Resistant to change" patient (Mann-Whitney U test: *p* = 0.05).

### 6.1.2 Patient dialogue act

The patient expresses significantly more intention to change unhealthy behaviors during the second half of the conversation than during the first half (T-test: *p* = .0005). There is also a difference in the use of changing statements according to at least two groups of patient types (Kruskal-Wallis H test: *p* = .03). Patients share more information during the first half of the conversation than during the second half (T-test: *p* = .002). Yet, there were no differences between patient types (Kruskal-Wallis H test *p* = .73).

## 6.2 Expressivity related behaviors

In this section, we analyze statistical differences in behaviors related to the expressivity according to different patient types and timing in the dialog.

### 6.2.1 Arm openness

Patients show significantly more arm openness during the second half than during the first half of the conversation (T-test: *p* = .04), while no significant differences were observed for the arm openness of the therapist between the first and the second half of the conversation (T-test: *p* = .45); considerable differences were observable between
patient types for the patient's body arm openness (Kruskal-Wallis H test: *p* < .000001). "Resistant to change" patients display more arm openness, while "Receptive" patients display smaller motions with less arm openness (Mann-Whitney U test: all *p* < .00001). There were no significant variations in arm openness between groups of talk type for patients (Kruskal-Wallis H test: *H*(2) = .06, *p* = .97) and therapists (Kruskal-Wallis H test: *H*(2) = 3.7, *p* = .16). Meaning that the expression of change talk or sustain talk does not have an effect on arm openness.

### 6.2.2 Loudness

There were variations in patient loudness between the start of the conversation and its ending (T-test: *p* = 0.003). Specifically, patients exhibit a higher voice volume towards the dialogue's end, although such a trend isn't mirrored in therapist loudness (T-test: *p* = 0.29 for patients). There were notable differences in loudness across various patient types for both patients and therapists (Kruskal-Wallis H test: *H*<sub>patient</sub>(2) = 53.18, *p*<sub>patient</sub> < .000001 and *H*<sub>therapist</sub>(2) = 58.9, *p*<sub>therapist</sub> < .000001). Both patients and therapists speak more loudly when the patient resists change and both adopt a softer tone when the patient is "Open to change" (Mann-Whitney U test: all *p* < .005). This difference is not significant between the therapists interacting with "Receptive" and "Resistant to change" patients (Mann-Whitney U test: *p* = 0.3) (see Fig. 5). Furthermore, loudness discrepancies related to expressed talk types emerge among patients (Kruskal-Wallis H test: *H*(2) = 23.47, *p* < .000001). Patients speak at a higher volume when expressing **sustain talk** than **change talk** and **neutral talk** (Mann-Whitney U test: all *p* < .006).

<figure><img src="image_9.png" /><figcaption>**Fig. 5:** Evolution of loudness separated by types. \* *p* < .1, \*\* *p* < .05, \*\*\* *p* < .001, \*\*\*\* *p* < .001, \*\*\*\*\* *p* < .0001</figcaption></figure>

## 6.3 Social related behaviors

In this section, we analyze statistical differences in behaviors related to social behaviors according to different types and timing in the dialog.
### 6.3.1 Therapist social dialogue acts

There were no significant differences in the use of empathic reactions between the first and second half of the conversation (T-test: *p* = .41). However, there is a difference between at least two groups of patients (Kruskal-Wallis H test: *p* = .04). The therapist tends to use more empathic reactions when interacting with a " receptive " patient than when interacting with a " resistant to change" patient (Mann-Whitney U test: *p* = .08).

### 6.3.2 Smiles

There were no significant differences in the number of therapists' smiles between the first and second half of the conversation (T-test: *p* = .4). However, their smiles last longer during the second half (T-test: *p* < .0001). In contrast, the patient smiles less during the second half of the interaction (T-test: *p* < .0001) but with more prolonged (T-test: *p* < .0001) and more intense smiles (T-test: *p* < .0001). There are differences between at least two groups of patients in smiles of patients and therapists for the number of smiles (Kruskal-Wallis H test: *p*< patient < .000001, *p*< therapist < .000001), duration (Kruskal-Wallis H test: *p*< patient < .000001, *p*< therapist < .000001) and intensity (Kruskal-Wallis H test: *p*< patient < .000001, *p*< therapist < .000001). Patients smile more when they are "Resistant to change" and less when they are "Open to change" (Mann-Whitney U test: all *p* < .000001). However, patients' smiles were longer when they are "Open to change" and shorter when they are "Resistant to change" (Mann-Whitney U test: all *p* < .01). Smiles were also significantly more intense when patients are "Resistant to change" and less intense when they are "Open to change" (Mann-Whitney U test: all *p* < .0001). Therapists smile more when patients they are interviewing are "Resistant to change" and less when they are "Open to change" (Mann-Whitney U test: all *p* < .0001). Smiles last longer when the patient is "Open to change" and shorter when they are "Receptive" (Mann-Whitney U test: all *p* < .05). Smiles are more intense when the patient is "Resistant to change" and less intense when the patient is "Open to change" (Mann-Whitney U test: all *p* < .0001). There are differences between at least two different types of patient talk for the therapists' and the patients' smiles in terms of number (Kruskal-Wallis H test: *p*< patient < .01, *p*< therapist < .0001), duration (Kruskal-Wallis H test: *p*< patient < .0001, *p*< therapist < .0001) and intensity (Kruskal-Wallis H test: *p*< patient < .0001, *p*< therapist < .0001). Patients smile more when they express **neutral talk** than when they listen (Mann-Whitney U test: *p* < .05). When the patient is listening, their smiles last significantly longer (Mann-Whitney U test: *p* < .0001), and their smiles are considerably more intense than when they have the speaking turn and express **neutral or sustain talk** (Mann-Whitney U test: *p* < .05). Therapists smile significantly more when they have the turn (Mann-Whitney U test: *p* < .0001); they also smile more when they listen to **neutral talk** than when they listen to **change talk** (Mann-Whitney U test: *p* = .0032). They also smile significantly longer(Mann-Whitney U test: *p* < .0001) and intensely (Mann-Whitney U test: *p* < .0001) when they have the turn.
## 6.4 Alignment

In this section, we analyze statistically significant differences in behaviors related to alignment according to different types and timing in the dialog.

### 6.4.1 Verbal alignment

The Dialign tools (Dubuisson Duplessis et al, 2021) provide for each utterance the list of verbal expressions initiated by a speaker that the other speaker reuses. Throughout the dialogue, there is a discernible upward trend in the recurrence of verbal expressions, both for therapist-initiated and patient-initiated utterances. There is a notable variance in repeated expressions based on the type of patient talk and the type of patient (Kruskal-Wallis H test: all *p* < .05). A higher frequency of expressions is reused when patients convey **neutral** statements (Mann-Whitney U test: all *p* < .0001). Additionally, the therapist reuses more of the patient's expressions when the patient is categorized as "Open to change" or "Receptive" (Mann-Whitney U test: all *p* < .0001).

### 6.4.2 Arm openness alignment

The alignment of the arm openness between the patients and the therapists is quantified by the absolute value of the difference between the therapist's and the patient's arm openness. This is a simplistic measure of arm openness, as it does not consider temporal dynamics. This limitation arises partly because only a limited number of frames show both speakers conjointly, and those frames are not sequentially continuous. There were statistical differences in alignment between at least two types of patients (Kruskal-Wallis H test: *H*(2) = 12, *p* = .002). "Receptive" patients were significantly more aligned with their therapists than "Resistant to change" patients (Mann-Whitney U test: *p* = .0000001) and "Open to change" patients (Mann-Whitney U test: *p* = .02). There were no differences between patient talk types (Kruskal-Wallis H test: *H*(2) = .19, *p* = .9).

# 7 Discussion

In this section, we discuss the consequences and interpretation of the differences found in the previous section.

## 7.1 Task related

Throughout the conversation, we observe an evolution of the dialogue acts used by the therapist and the patient (RQ1). During the first half of the conversation, the therapist asks many questions to assess the context and concerns of the patient. In contrast, the patient shares personal information to answer those questions. During the second half of the conversation, the therapist makes more statements about planning and advises the patients on addressing their problems. In response, the patient has more intention to change, reacting to the therapist's input. However, this global evolution can vary depending on the type of patient (RQ2). The therapist uses more invitation to shift outlook when the patient is classified as "Receptive" than when they are
classified as "Resistant to change". This may imply that therapists attempt to pro-
vide new perspectives to patients who appear open to receiving them but have not yet
determined their motivation to change (namely, the "Receptive" ones). Furthermore,
patients classified as "Resistant to change" tend to manifest behaviors indicative of
a reluctance to engage in dialogue and use significantly fewer statements explicitly
directed toward changing their behaviors.

## 7.2 Behavior expressivity

We observe an evolution in the behavior expressivity of the patients throughout the conversation (RQ1). Indeed, patients tend to be physically more expressive in the second half of the conversation, where they talk more loudly and display behavior with a larger arm openness. These observable changes suggest a growing confidence and self-assurance as the dialogue unfolds. Differences in this degree of expressivity were also observable between patient types (RQ2). In particular, "Resistant to change" patients exhibit a more pronounced behavior expressivity, characterized by louder speech and a more openly engaged demeanor than other patient types. This observation implies that these patients were significantly more assertive and self-assured about their viewpoints (Vickers, 2010). Similarly, patients were louder when they perform sustain talk (RQ3). An interpretation of these results is that patients who are "Resistant to change" or expressing sustain talk when faced with an MI therapist are confident in their beliefs (Scherer et al, 1973), which might explain why it is more difficult to change their minds.

## 7.3 Social behaviors

Social behaviors such as smiles or expressions of empathy are indicators of engagement in conversation (Corrigan et al, 2016) and are often associated with positive therapy outcomes (Jani et al, 2012). Our results show that the interpersonal relations between the patient and the therapist improve significantly as the interaction progresses (RQ1). The duration and intensity of smiles from both the patient and the therapist increase in the second half of the interaction, highlighting the progression of their relationship throughout the conversation.

We also observe variations in the expression of these behaviors depending on the patient's type of talk (RQ3). When expressing **neutral talk**, patients exhibit longer-lasting smiles. In MI dialogue, **neutral talk** refers to discussions outside the scope of behavior change, including small talk used to build the relationship between the patient and the therapist, which can explain the increase in social behaviors. Furthermore, when the patient expresses change talk, which aligns with the therapist's goal, therapists are significantly more positive compared to when they express sustain talk. This aligns with the notion that building a relationship is easier when both parties share a similar goal.

Social behaviors also vary between patient types (RQ3). Therapists tend to use
more empathic statements with "Receptive" patients compared to those who are
"Resistant to change." This may stem from the need to build a relationship with resis-
tant patients to find what can motivate them to change. "Open to change" patients
display a more positive disposition, with longer-lasting smiles, suggesting a natural alignment with the therapeutic process. "Resistant to change" patients tend to smile more intensely and frequently. When patients are resistant to change, therapists also smile more and more intensely, indicating an effort to foster interpersonal relations with resistant patients, likely to enhance the therapeutic outcome.

## 7.4 Alignment

The alignment between the therapist and the patient, computed through their verbal and arm openness alignment, tends to grow throughout the conversation (RQ1). More expressions are reused in the second half than in the first half. This alignment is also more potent when the patient is "Open to change" and "Receptive" (RQ2). The arm openness of the patient's and the therapist is also more aligned when the patient is "Open to change" and "Receptive", and more expressions are reused. Such alignment emerges as the goals of the therapist and the patient are also more aligned. A high verbal alignment can also be a sign of engagement (Becker, 2006). These results can, therefore, be a sign that "Open to change" and "Receptive" patients are more engaged in the conversation with the therapist as their initial goal (changing their behavior) is validated and encouraged by the conversation. We have computed the alignment of the arm openness of both interlocutors as a global value, that is, we have only computed the difference between the two interlocutors' openness over the whole interaction, see Section 6.4.2. Considering temporal information would provide more refined information. However, it would require taking into account that this type of alignment may not be synchronous and would require computing it at different levels of coordination Schmidt and Richardson (2008).

## 8 Conclusion

This paper presents EMMI, composed of multimodal features extracted from two publicly available corpora, AnnoMI and MID. We analyze these features to extract pertinent behaviors for developing a virtual agent performing motivational interviews, emphasizing social behaviors. We highlight differences in behavior for both the therapist and the patient during the conversations, and depending on the **change talk** conducted by the patient. Our analysis shows that patients can be clustered into three groups, "Open to change," "Resistant to change," and "Receptive", whose behaviors differ significantly. The therapist also exhibits different behaviors depending on the type of patient. This underscores the importance of enabling virtual interviewers to adapt their behavior based on the patient's type and current state.

Some limitations of this work are that the studied videos portrayed actors with no indication of how they were instructed and whether the videos were scripted. The videos are also not all in the same settings, and some nonverbal data are missing due to the positioning of the camera, which limited our ability to study certain nonverbal behaviors. The features are extracted automatically, which presents a limitation. However, our ultimate goal is to create an automatic MI virtual agent, meaning the features will be extracted automatically in that context as well; hence, we need similar data to make meaningful comparisons. Another limitation is that our analysis is
based on single-session videos, whereas MI typically occurs across multiple sessions. This constraint stems from the fact that the available corpora only include recordings of individual sessions. We also do not take into account demographic information or other interesting nonverbal behaviors such as interruptions, iconic gestures, and self-repair. Nevertheless, this study provides insights into patient behavior that can be useful in the future.

In future work, identifying and characterizing these patient types could serve as a foundation for developing a multimodal dialogue system to deliver customized MI interventions. The insights gained from understanding how patients and therapists evolve in their communication style, behavior expressivity, social and empathic behaviors, and alignment throughout the conversation can inform the design of an intelligent virtual MI interviewer. This interviewer could dynamically adapt its communication strategies, voice, and content based on the identified type of patient, thus enhancing the effectiveness of MI.

## Acknowledgment

This work was partially funded by the ANR-DFG-JST Panorama and ANR-JST-CREST TAPAS (19-JSTS-0001-01) projects.

## Ethical Impact Statement

While the primary aim of our research is not to replace therapists but rather to extend support and accessibility to underserved populations, the personalized adaptability we propose could inadvertently open avenues for manipulation. Introducing an empathetic virtual therapist also raises concerns regarding human-agent attachment, which necessitates careful consideration. In our final application, we are committed to transparently informing participants that they are engaging with a virtual therapist, one that is fallible, may make mistakes, and lacks emotional capacity. It is crucial to emphasize that the agent is not a substitute for human-to-human therapy.

The features are extracted from videos sourced from YouTube. The videos feature actors and do not include personally identifiable information, thus safeguarding privacy. While the use of actors may introduce a bias, it also allows for privacy protection.

Acknowledging that our data primarily consists of interactions with American participants introduces a potential bias in dialogue acts classification results, particularly concerning vocabulary usage. To mitigate this risk, we aim to incorporate examples from other dialects, such as British or African-American Vernacular English, to enhance the classifier's adaptability across diverse linguistic contexts. The proposed dialogue act classifier will be used as a natural language understanding module within human-agent conversations. However, it is imperative to recognize that automatic classifications can yield errors, particularly in the sensitive context of therapy, where mistakes can have significant repercussions. Consequently, outputs from the classifier should be handled with utmost care. This is also true of the proposed separation of patients into types. In contrast to prevailing methodologies, we have opted to utilize the open-source Mistral model instead of OpenAI's GPT. This enables cost-free usage,
open-source availability, and offline deployment. Given the importance of maintaining patient privacy in the context of utterance classification in therapy, the ability to operate the model offline is essential.

All annotations and code are publicly available to facilitate reproducibility.

## References

Angus LE, Kagan F (2009) Therapist empathy and client anxiety reduction in motivational interviewing: "she carries with me, the experience". Journal of Clinical Psychology 65(11):1156-1167

Baltrusaitis T, Zadeh A, Lim YC, et al (2018) Openface 2.0: Facial behavior analysis toolkit. In: 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), IEEE, pp 59-66

Beauvais C (2019) Motivational interviewing to improve treatment adherence

Becker HS (2006) Sur le concept d'engagement. SociologieS

Bickmore TW, Pfeifer LM, Jack BW (2009) Taking the time to care: empowering low health literacy hospital patients with virtual nurse agents. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 1265-1274

Bickmore TW, Kimani E, Trinh H, et al (2018) Managing chronic conditions with a smartphone-based conversational virtual agent. In: Proceedings of the 18th International Conference on Intelligent Virtual Agents, pp 119-124

Cameron G, Cameron D, Megaw G, et al (2017) Towards a chatbot for digital counselling. In: Proceedings of the 31st International BCS Human Computer Interaction Conference (HCI 2017) 31, pp 1-7

Campano S, Durand J, Clavel C (2014) Comparative analysis of verbal alignment in human-human and human-agent interactions. In: LREC, Citeseer, pp 4415-4422

Cao Z, Simon T, Wei SE, et al (2017) Realtime multi-person 2d pose estimation using part affinity fields. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7291-7299

Chiu YY, Sharma A, Lin IW, et al (2024) A computational framework for behavioral assessment of llm therapists. arXiv preprint arXiv:240100820

Corrigan LJ, Peters C, Küster D, et al (2016) Engagement perception and generation for social robots and virtual agents. Toward Robotic Socially Believable Behaving Systems-Volume I: Modeling Emotions pp 29-51

Creber RM, Patey M, Lee CS, et al (2016) Motivational interviewing to improve self-care for patients with chronic heart failure: Miti-hf randomized controlled trial.
Patient education and counseling 99(2):256-264

Decker SE, Nich C, Carroll KM, et al (2014) Development of the therapist empathy scale. Behavioural and cognitive psychotherapy 42(3):339-354

Denecke K, Vaaheesan S, Arulnathan A (2020) A mental health chatbot for regulating emotions (sermo)-concept and usability test. IEEE Transactions on Emerging Topics in Computing 9(3):1170-1182

Dubuisson Duplessis G, Langlet C, Clavel C, et al (2021) Towards alignment strategies in human-agent interactions based on measures of lexical repetitions. Language Resources and Evaluation 55:353-388

Efron B, Tibshirani RJ (1994) An introduction to the bootstrap: Crc press. Ekman, P, & Friesen, WV (1978) Manual for the facial action coding system

Eyben F, Wöllmer M, Schuller B (2010) Opensmile: the munich versatile and fast open-source audio feature extractor. In: Proceedings of the 18th ACM international conference on Multimedia, pp 1459-1462

Fiske A, Henningsen P, Buyx A (2019) Your robot therapist will see you now: ethical implications of embodied artificial intelligence in psychiatry, psychology, and psychotherapy. Journal of medical Internet research 21(5):e13216

Fitzpatrick KK, Darcy A, Vierhile M (2017) Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (woebot): a randomized controlled trial. JMIR mental health 4(2):e7785

Friesen E, Ekman P (1978) Facial action coding system: a technique for the measurement of facial movement. Palo Alto 3(2):5

Galland L, Pelachaud C, Pecune F (2024) Seeing and hearing what has not been said. IEEE, pp 1-9

Gillam DG, Yusuf H (2019) Brief motivational interviewing in dental practice. Dentistry Journal 7(2):51

Jani BD, Blane DN, Mercer SW (2012) The role of empathy in therapy and the physician-patient relationship. Forschende Komplementärmedizin/Research in Complementary Medicine 19(5):252-257

Jiang AQ, Sablayrolles A, Mensch A, et al (2023) Mistral 7b. arXiv preprint arXiv:231006825

Lord SP, Sheng E, Imel ZE, et al (2015) More than reflections: Empathy in motivational interviewing includes language style synchrony between therapist and client. Behavior therapy 46(3):296-303
Magill M, Gaume J, Apodaca TR, et al (2014) The technical hypothesis of motivational interviewing: A meta-analysis of mi's key causal model. Journal of consulting and clinical psychology 82(6):973

Magill M, Apodaca TR, Borsari B, et al (2018) A meta-analysis of motivational interviewing process: Technical, relational, and conditional process models of change. Journal of consulting and clinical psychology 86(2):140

Malhotra G, Waheed A, Srivastava A, et al (2022) Speaker and time-aware joint contextual learning for dialogue-act classification in counselling conversations. In: Proceedings of the fifteenth ACM international conference on web search and data mining, pp 735-745

Mercado J, Espinosa-Curiel IE, Martínez-Miranda J (2023) Embodied conversational agents providing motivational interviewing to improve health-related behaviors: Scoping review. Journal of Medical Internet Research 25:e52097

Miller WR, Moyers TB (2017) Motivational interviewing and the clinical science of Carl Rogers. Journal of consulting and clinical psychology 85(8):757

Miller WR, Rollnick S (2012) Motivational interviewing: Helping people change. Guilford press

Miller WR, Moyers TB, Ernst D, et al (2003) Manual for the motivational interviewing skill code (misc). Unpublished manuscript Albuquerque: Center on Alcoholism, Substance Abuse and Addictions, University of New Mexico

Moyers TB, Miller WR, Hendrickson SM (2005) How does motivational interviewing work? therapist interpersonal skill predicts client involvement within motivational interviewing sessions. Journal of consulting and clinical psychology 73(4):590

Nakano YI, Hirose E, Sakato T, et al (2022) Detecting change talk in motivational interviewing using verbal and facial information. In: Proceedings of the 2022 International Conference on Multimodal Interaction, pp 5-14

Pérez-Rosas V, Mihalcea R, Resnicow K, et al (2016) Building a motivational interviewing dataset. In: Proceedings of the Third Workshop on Computational Linguistics and Clinical Psychology, pp 42-51

Pérez-Rosas V, Mihalcea R, Resnicow K, et al (2017a) Understanding and predicting empathic behavior in counseling therapy. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp 1426-1435

Pérez-Rosas V, Mihalcea R, Resnicow K, et al (2017b) Predicting counselor behaviors in motivational interviewing encounters. In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1,
Pérez-Rosas V, Wu X, Resnicow K, et al (2019) What makes a good counselor? learning to distinguish between high-quality and low-quality counseling conversations. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp 926–935

Rauzy S, Goujon A (2018) Automatic annotation of facial actions from a video record: The case of eyebrows raising and frowning. In: Workshop on" Affects, Compagnons Artificiels et Interactions", WACAI 2018, pp 7–pages

Ring L, Bickmore T, Pedrelli P (2016) An affectively aware virtual therapist for depression counseling. In: ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) workshop on Computing and Mental Health, pp 01951–12

Sakoe H, Chiba S (1978) Dynamic programming algorithm optimization for spoken word recognition. IEEE transactions on acoustics, speech, and signal processing 26(1):43–49

Scherer KR, London H, Wolf JJ (1973) The voice of confidence: Paralinguistic cues and audience evaluation. Journal of Research in Personality 7(1):31–44

Schmidt RC, Richardson MJ (2008) Dynamics of interpersonal coordination. In: Coordination: Neural, behavioral and social dynamics. Springer, p 281–308

for Substance Abuse Treatment C, et al (2019) Enhancing motivation for change in substance use disorder treatment. Treatment improvement protocol (TIP) series

Tavabi L, Stefanov K, Zhang L, et al (2020) Multimodal automatic coding of client behavior in motivational interviewing. In: Proceedings of the 2020 International Conference on Multimodal Interaction, pp 406–413

Thorndike RL (1953) Who belongs in the family? Psychometrika 18(4):267–276

Vickers E (2010) The loudness war: Background, speculation, and recommendations. In: Audio Engineering Society Convention 129, Audio Engineering Society

Wu Z, Balloccu S, Kumar V, et al (2022) Anno-mi: A dataset of expert-annotated counselling dialogues. In: ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp 6177–6181

Wu Z, Balloccu S, Kumar V, et al (2023) Creation, analysis and evaluation of annomi, a dataset of expert-annotated counselling dialogues. Future Internet 15(3):110

Younsi N, Pelachaud C, Chaby L (2024) Beyond words: Decoding facial expression dynamics in motivational interviewing. In: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp 2365–2374