# Exploring emotion expression recognition in older adults interacting with a virtual Coach

Cristina Palmero, Mikel Develasco, Mohamed Amine Hmani, Aymen Mtibaa, Leïla Ben Letaifa, Pau Buch-Cardona, Raquel Justo, Terry Amorese, Eduardo González-Fraile, Begoña Fernández-Ruanova, et al.

► To cite this version:

Cristina Palmero, Mikel Develasco, Mohamed Amine Hmani, Aymen Mtibaa, Leïla Ben Letaifa, et al. Exploring emotion expression recognition in older adults interacting with a virtual Coach. IEEE Transactions on Affective Computing, 2025, 16 (3), pp.2303 - 2320. 10.1109/taffc.2025.3558141. hal-05280245

HAL Id: hal-05280245

https://hal.science/hal-05280245v1

Submitted on 24 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
Exploring Emotion Expression Recognition in
Older Adults Interacting with a Virtual Coach

Cristina Palmero, Mikel deVelasco, Mohamed Amine Hmani, Aymen Mtibaa, Leila Ben Letaifa, Pau Buch-Cardona, Raquel Justo, Terry Amorese, Eduardo González-Fraile, Begoña Fernández-Ruanova, Jofre Tenorio-Laranga, Anna Torp Johansen, Micaela Rodrigues da Silva, Liva Jenny Martinussen, Maria Stylianou Korsnes, Gennaro Cordasco, Anna Esposito, Mounim A. El-Yacoubi, Dijana Petrovska-Delacrétaz, M. Inés Torres, and Sergio Escalera

***Abstract***—The EMPATHIC project aimed to design an emotionally expressive virtual coach capable of engaging healthy seniors to improve well-being and promote independent aging. In particular, the system's human sensing capabilities allow for the perception of emotional states to provide a personalized experience. This paper outlines the development of the emotion expression recognition module of the virtual coach, encompassing data collection, annotation design, and a first methodological approach, all tailored to the project requirements. With the latter, we investigate the role of various modalities, individually and combined, for discrete emotion expression recognition in this context: speech from audio, and facial expressions, gaze, and head dynamics from video. The collected corpus includes users from Spain, France, and Norway, and was annotated separately for the audio and video channels with distinct emotional labels, allowing for a performance comparison across cultures and label types. Results confirm the informative power of the modalities studied for the emotional categories considered, with multimodal methods generally outperforming others (around 68% accuracy with audio labels and 72-74% with video labels). The findings are expected to contribute to the limited literature on emotion recognition applied to older adults in conversational human-machine interaction, and guide the development of future systems.

# I. INTRODUCTION

EMOTION recognition plays a pivotal role in conversational human-machine interaction (HMI) [1], enabling systems to perceive and respond to users' emotional states [2], [3]. Research in affective computing has long proved the possibility of detecting emotion expressions with data-driven approaches using different input modalities, mainly linguistic, acoustic, and facial expressions [4], [5]. Multimodal approaches have also shown promising results in enhancing recognition accuracy and robustness [6]. The literature has shifted from recognizing acted, contextless prototypical expressions to more spontaneous reactions and in-the-wild data,

C. Palmero, P. Buch-Cardona, and S. Escalera are with Universitat de Barcelona and Computer Vision Center, Spain. E-mail: crpalmec7@alumnes.ub.edu.

M. deVelasco, L. Ben Letaifa, R. Justo, and M.I. Torres are with Universi-
dad del País Vasco UPV-EHU, Spain.

M.A. Hmani, A. Mtibaa, D. Petrovska-Delacrétaz, and M.A. El-Yacoubi are
with SAMOVAR, Telecom SudParis, Institut Polytechnique de Paris, France.

T. Amorese, G. Cordasco, and A. Esposito are with Università degli Studi
della Campania L. Vanvitelli, Italy.

E. González-Fraile is with Universidad Internacional de La Rioja, Spain.

B. Fernández-Ruanova and J. Tenorio-Laranga are with Osatek, Spain.

A. Torp Johansen, M. Rodrigues da Silva, L. J. Martinussen, and M.
Stylianou Kornes are with Oslo University Hospital, Norway.

presenting numerous challenges for unimodal and multimodal
models [6]. In addition to the increased appearance and
behavioral variability, spontaneous emotions are more sub-
tle and difficult to disambiguate, significantly differing from
acted emotions in surface representation [7]. Consequently,
the emotional space is smaller and more compact [8], [9].
Furthermore, natural contexts suffer from a high category
imbalance, negatively affecting the learning process of data-
driven approaches. HMI users tend to exhibit less intense
emotional responses than when interacting with other humans
due to the limited emotional capacity of artificial agents,
resulting in more neutral expressions [9]. Visual-based recog-
nition is further affected by the *speaking effect*, for which
facial deformations caused by speaking can be mistaken for
emotional expressions [10]. Lastly, establishing a reliable gold
standard remains difficult due to subjective perceptual anno-
tation procedures, resulting in low agreement and potential
discrepancies between emotions expressed and perceived [11].

The global aging of the population is generating new socioeconomic challenges. Thus, older adults have been positioned as a clear beneficiary of technological development in HMI [2], [12]–[14]. Computational emotion recognition research has traditionally examined emotional expressions in the general population, without targeting a specific group. As such, older adults have been usually underrepresented given the considered cohorts. However, understanding emotions in this age group is of special importance, as it can lead to the development of customized virtual coaching applications and companions, and healthcare technologies that foster active aging and independent living. This presents additional computational challenges, since aging changes facial features, voices, and speaking styles, among others [15], [16]. For example, a higher intensity in facial expressions and speech is associated with a higher emotion recognition accuracy; however, older subjects display less intense vocal and facial expressions compared to younger subjects [17], [18]. Therefore, models trained in other age groups do not perform optimally for recognition in older adults [18], and models trained on data from this age group tend to show lower performance than other groups [19], [20]. The lack of public databases including older adults hinders progress on this front. To our knowledge, only two datasets that focus on older adults provide non-acted data: the ComParE Elderly Emotion Sub-Challenge speech dataset [21], which includes personal narratives; and ElderRe-
act [18], containing monologue videos of older adults reacting
to specific items spontaneously, but possibly exaggerating their
responses due to the nature of the videos. Very few non-acted,
interaction-oriented datasets include older adults [22], and we
are unaware of any HMI dataset featuring them.

This paper presents the first comprehensive study on com-
putational, non-verbal discrete emotion expression recognition
in interactions between older adults and a simulated Virtual
Coach (VC), as a specific case of HMI scenario. The work was
developed as part of the European EMPATHIC project [23],
[24], which aimed to explore and validate new interaction
paradigms for empathic, expressive, and advanced VCs to
improve independent, healthy-life-years of this age group. As
part of the project, 157 participants over 65 years old from
three countries (Spain, France, and Norway) were recorded
interacting with an initial version of the EMPATHIC-VC in
a Wizard of Oz (WoZ) paradigm. This data aided in system
development and in the study of the interaction between older
adults and VCs<fnref n="1" />. Under this framework, we first describe the
emotion annotation procedure and methodological choices tai-
lored to the project requirements. Then, using a deep learning-
based approach, we investigate the contribution of different
modalities for emotion recognition, including speech, facial
expressions, eye gaze, and head dynamics, individually and
combined, in various evaluation scenarios.

This framework provides two unique features that have re-
ceived limited attention [22], which we exploit in our work as
additional contributions. First, it allows us to perform a com-
parative analysis across cultures (where culture influences not
only emotional expression but also the annotation procedure)
and languages as well as multi-country versus country-specific
training. Second, the project involved a channel-specific anno-
tation, providing two distinct sets of emotional labels (audio-
and video-based). Thus, we consider speech from audio and
facial expressions from video as main modalities, while less-
explored eye gaze and head movement cues act as additional
modalities that can also be extracted from video. We assess
the effectiveness of the main modalities in recognizing their
associated labels, and the possible performance improvement
when being combined with the remaining (auxiliary) modal-
ities. Additionally, we conduct a cross-channel evaluation,
wherein the main modality of one label type and the auxiliary
modalities are individually employed to recognize the labels
derived from the other channel. This offers insight into the
transferability and adaptability of modalities across label types,
possibly revealing shared emotional cues. Lastly, we analyze
performance differences between training and evaluating on
spoken and silent instances, to understand how the presence
or absence of speech affects performance for this age group.

This paper is organized as follows. Sec. II reviews current
trends in emotion recognition with the modalities considered
in this work. Sec. III describes the corpus and the annotation
protocol. Sec. IV details our computational approach. Sec. V
describes the evaluation protocol and results, which are dis-
cussed in Sec. VI. Finally, Sec. VII concludes the paper.

II. RELATED WORK

A. *Models of emotion*

Expressions of emotion are generally represented by two
main different models: a categorical or discrete model, and
a dimensional or continuous model. The categorical model
identifies a set of discrete emotional categories, ranging from
the basic Ekman emotions [25] (happy, surprised, contempt,
sad, fearful, disgusted, and angry), to a larger set with more
specific and realistic affective states. Indeed, ordinary commu-
nication involves a variety of complex feelings that cannot be
characterized by a reduced, fixed set of categories [26].There-
fore, such categories are usually selected considering the task
at hand. For instance, categories such as bored, frustrated,
delighted, calm, satisfied, or excited are more applicable to
HMI scenarios than most basic states [27].

Given the complexity of the emotional semantic space,
a number of researchers [26], [28] are more in favor of
adopting a dimensional model such as the circumplex model
of affect [29]. In the dimensional model, each affective state is
represented by a point in a two-dimensional space, where the
valence dimension represents the polarity of the emotion, i.e.,
a positive or negative value along a continuum, and the arousal
dimension represents the degree of emotional activation. i.e.,
values vary from low to high along a continuum. Other ver-
sions include a third dimension, dominance, which represents
a sense of control over the situation while experiencing the
emotion. The Valence, Arousal, and Dominance (VAD) model
has been widely exploited for audio/video-based emotion
recognition [9], [26], [30], allowing for the encoding of slight
emotional changes over time [30].

Both models have advantages and drawbacks. For instance,
emotional categories may overlook intensity and exhibit fuzzy
boundaries. Conversely, dimensional models introduce more
subjectivity in emotion scaling across raters. Ultimately, the
choice depends on the objectives of the task. In our case,
we are interested in detecting key emotional events during
interactions, enabling the EMPATHIC-VC system to react and
adapt to them in a practical and interpretable way. Thus, the
categorical model better fits the needs of the system.

B. *Emotions from speech*

The speech signal captures the speaker's communicative
intention, encompassing not only the words spoken but also the
intonation, prosody, pauses, and other paralinguistic elements
that contribute to the message. In the same way, speech
provides a lot of information about the speaker, their accent,
profile, speaking style, and current emotional state [31], [32].

The most commonly used features for speech emotion
recognition (SER) are based on Low-Level Descriptors
(LLDs), such as zero crossing rates, pitch, formants, energy,
jitter, shimmer, spectral centroids, Mel-Frequency Cepstral
Coefficients (MFCC), flux, etc., as well as on their descriptive
statistics or functionals (e.g., mean, SD, quartiles) [9], [31].
Some works have proposed the standardization of the feature
sets. However, only GeMAPS, which contains a combination
of the previously mentioned LLDs and functionals, and the
feature sets proposed in the ComPaRE challenge series, which

<footnote n="1">¹The EMPATHIC data corpus can be found in the European Language Resources Association (ELRA) catalogue (http://catalogue.elra.info/en-us/): Corpus ISLRN: 631-345-309-445-9 and ELRA ID: ELRA-S0414.</footnote>
are variations of GeMAPS, have become a reference [33], [34]. Spectrograms have also been used as a sequence of features represented as an image, which has been demonstrated to be specifically useful to feed Convolutional Neural Networks (CNNs) [32]. More recently, the first framework for self-learning rich representations of speech was published, called Wav2Vec [35], which was initially used for SER in English by [36] and in Spanish by [9]. Shortly afterward, new frameworks were proposed, including Hubert [37], UniSpeech [38], and WavLM [39] (see [32] for a comparison).

Similarly to other domains, the rise of deep learning
also caused a gradual transition from traditional classifiers
to deep neural networks (DNNs) for SER [40], [41]. Current approaches feature Multilayer Perceptrons (MLP) [32], CNNs [9], Recurrent Neural Networks [32], Transformers [42], and their combinations [9], [43].

C. *Emotions from facial expressions*

Facial expressions are considered one of the most significant
means for humans to express their emotions and intentions in
their daily communication [25]. Facial expression recognition
(FER) systems can be divided into two main categories:
static-image FER and dynamic-sequence FER [44]. In static-
based methods, the feature representation is based only on the
spatial information associated with a single image, whereas
dynamic-based methods consider the temporal relation among
contiguous frames as well as the facial deformation dynamics.

In turn, and similarly to SER, FER approaches can also be
divided into conventional and deep learning based approaches.
The former is usually composed of three major steps: face
and landmarks detection, feature extraction, and emotion clas-
sification. These conventional algorithms usually extract face-
based handcrafted features such as Gabor filters [45], local
binary patterns [46], and histograms of oriented gradients [47].
However, they often lack enough generalizability in in-the-
wild settings. By contrast, deep learning-based approaches
are used as a conjoint feature extraction tool and facial ex-
pression classifier, reducing the dependency on preprocessing
techniques and human expertise-based feature extraction. The
same neural network approaches discussed for SER have
been applied to FER with similar results; hence, we avoid
repeating them here. We refer the reader to the surveys of [44],
[48], [49] for a comprehensive review of the state of the
art. As an example of approach related to the one used in
this paper, we highlight the work of [50], one of the first
to demonstrate the capability of CNNs to recognize Ekman
emotions by outperforming traditional methods on popular
posed and spontaneous expression datasets.

D. *Emotions from eye gaze and head pose*

Extensive behavioral and neuroscience literature has con-
firmed a relationship between eye state, gaze direction, and
facial expressions on the perception of emotions and mental
states [51]. Eye-related features that have been studied or
used the most in affective computing are: pupil size, blinks,
gaze direction, direct/averted gaze, extracted patterns of gaze
events, and eye aperture/closure [52], [53]. Dedicated eye

trackers are generally required to extract these features with
high accuracy. However, this is impractical for many every-
day scenarios or HMI settings, such as the EMPATHIC-VC,
where a non-obtrusive or lower-cost approach is preferred.
For such scenarios, regular cameras can now be used to
estimate eye gaze and approximate the location of pupil
and eye landmarks by means of appearance- or model-based
methods. In particular, appearance-based gaze estimation has
improved significantly during the past decade, boosted by deep
learning advances [54]. Therefore, a number of works compute
statistical features, or functionals, from the raw or smoothed
estimated gaze trajectories over a time window, or compute
features (e.g., eye closure, pupil size) based on specific eye
landmarks instead [53], [55]. Blinks can usually be detected
via dedicated appearance-based methods [56], or by detecting
the action unit (AU) #45.

On a related note, there is evidence of the relationship
between head pose dynamics and expression and perception
of different emotional and mental states [57]–[59], being
particularly related to emotional intensity. Affect recognition
works have relied on head pose categorizations such as tilts,
nods, and shakes [57], [60], which usually require specific
action detectors. Recent approaches directly use temporal
3D rotational angles (yaw, pitch, and roll) to describe head
motion trajectories, as well as angular displacement, velocity,
acceleration, and window-based functionals computed from
such trajectories [61], dynamic features based on the Fourier
transform [62], or clustered sequences of kinemes [63]. Head
orientation is generally extracted with appearance-based meth-
ods and model-based 3D head registration [64].

Due to their relationship, a handful of works have combined
head and gaze features together for emotion recognition [65].
Although these features have been proven to be sufficient for
specific affective states in some scenarios [63], [66], they
are usually added to facial or speech modalities to provide
complementary rather than redundant information.

E. *Multimodal emotion expression recognition*

With significant advancements in multimodal machine
learning, multimodal emotion recognition has gained con-
siderable momentum lately (see [67], [68] for exhaustive
surveys on multimodal machine learning, and [4]–[6], [11] for
multimodal emotion recognition). Multimodal fusion methods
are broadly classified into feature-based, decision-based, and
hybrid approaches. The former consists in combining the
features extracted from different modalities, with methods
that range from naive feature concatenation to attention-based
approaches. Feature-based fusion allows learning from cross-
modal correlations; however, an alignment among modalities
is required since they may have different sampling rates or
representations (e.g., video frames vs audio segments), and not
all modalities may be available at all times. Instead, decision-
based fusion combines the scores or predictions of unimodal
models for a final multimodal prediction, thus alleviating
the alignment and incomplete data problems but disregarding
cross-modal correlations. Lastly, hybrid approaches combine
feature- and decision-based fusion. Generally, the best fusion
type is task- and dataset-dependent.
Most multimodal emotion recognition works combine at least paralinguistic and facial expression features [69], or acoustic and linguistic [43]. Gaze and head cues are usually combined with other features like facial information [70], [71] and/or speech cues [72], [73]. Nonetheless, their use is less explored compared to audiovisual fusion. One of the few works that combine speech, facial expressions, and gaze features is that of [55], which uses audio GeMAPS features, and a subset of the gaze functionals proposed by [66] and facial features extracted from a pre-trained CNN from video.

## F. Emotion expression recognition for older adults

The literature on audiovisual emotion recognition for older adults is limited. First attempts focused on unimodal models with handcrafted features [19], [20]. The ComParE challenge recently drew attention to the topic, in which challenge participants could use acoustic and linguistic features [21]. Concurrent to our work, [74] propose integrating facial expressions and speech sentiment in socially assistive robots for older adults. In particular, they use a static-based FER approach coupled with the sentiment from speech transcripts, combined in a decision-based fusion fashion.

However, the work of [18] is one of the few addressing discrete emotion recognition using the modalities considered in this work for such age group, based on ElderReact. More specifically, they extract gaze and head features, facial AUs (including blink), and facial landmarks from video, and voice quality, MFCCs, and prosody features from audio. Features are extracted per frame, and the mean and standard deviation (SD) of each feature computed for the entire video clip, with a single prediction per clip. Using the same dataset, [75] compared the performance of the extracted audio features to a spectrogram, and the extracted video features further processed with a CNN to a CNN trained on the raw video data. Results showed better performance for the extracted audio features and the raw video for most emotions considered. Finally, [76] outperformed the results of [18] by a large margin using only the visual modality with a Siamese network and a contrastive loss.

# III. EMPATHIC WOZ CORPUS

Here, we describe the subset of the EMPATHIC WoZ Corpus considered for this work, and the protocol followed for the annotation of audio and facial expressions from video.

## A. Data collection

The target population of the EMPATHIC project was defined as healthy older adults based on the following inclusion criteria: 1) above the age of 65 or turning 65 in 2019; 2) good hearing and sight (with or without glasses/hearing aid); 3) living independently at home; and 4) read, write, and speak the testing language fluently. Recruitment<fnref n="2" /> involved participants from Spain, France and Norway. A total of 157 participants

<figure><img src="image_2.png" /><figcaption>Fig. 1: Setup with a participant during an interaction session.</figcaption></figure>

(105 female) were recruited and participated in the first recording sessions of the project, of which 153 are included in the corpus. Participants are distributed as follows: 78 Spanish (54 female, mean age 69.5), 44 French (28 female, mean age 73.5), 31 Norwegian (21 female, mean age 74.8). The overall mean age was 71.8 years (SD=±6.8). All participants were properly informed and signed an informed consent prior to enrolling on the study. Hereinafter, we refer to the Spanish subset as SP, the French as FR, the Norwegian as NO, and the complete data as WHOLE (or WH).

We used the WoZ paradigm for data acquisition, commonly used when building systems based on natural language and other artificial intelligence-driven applications [77]. Its key principle is that study participants believe they are interacting with an autonomous system, while actually the actions of the system are controlled by a human (i.e., the *wizard*). This wizard is usually in a different room and connected to the study setting through a network connection. The interaction sessions combined different questionnaires and interaction with the EMPATHIC-VC, detailed in [2]. The setup consisted of a computer equipped with a webcam, a microphone, and an Internet connection (see Fig. 1). At the start of the session, participants chose one of five available visual representations of agents for their VC session. During the interaction, participants were alone with the VC to avoid bias or undesired interactions with the supervisor. Two dialogues of 5–10 min each were completed. The first served as an introduction to the system and thus did not focus on any specific issues. The second focused on the user’s nutrition habits and goals.

## B. Definition of labels

Our emotional labels correspond to the users' perceived expression of emotion. The procedure for selecting such emotion categories followed a three-step data-driven approach.

First, we considered the 27 categories defined in [78], which are based on the self-reported emotional states elicited by around 200 short videos over a population of nearly 1000 people. The list defines a rich semantic space of emotions, which includes categories such as *amusement* that were found to capture well the subjective emotional experience.

As a second step, we removed the categories that were highly unlikely to be encountered during the interaction between the user and the EMPATHIC-VC, and added some labels that might potentially be perceived in these interactions. We worked with the target languages simultaneously, i.e., Spanish,

<footnote n="2">²This protocol was approved by appropriate Institutional Review Boards of Spain (Ethical and Scientific Research Committee of University of the Basque Country -UPV/EHU. Cod: PI2018152), France (Commission Nationale de l'Informatique et des Libertés-CNIL- Cod: 2182146), and Norway (Ethical and Scientific Research Committee of the Oslo University Hospital).</footnote>
French, and Norwegian, to provide accurate terms to express the same feelings in different languages, considering that cultural context can be accounted for by the translation that the native speakers of each language can provide relative to the *Lingua Franca* (English in our case). The selected 18 labels were: *relieved, bored, excited, calm, sad, amused, puzzled, pleased, interested, tense, surprised, concerned, enthusiastic, skeptical, embarrassed, tired, delighted, and annoyed*.

Finally, we ran a set of pilot experiments on SP. Our goal was two-fold: 1) shorten the previous list by only considering the subset of emotions perceived during the interaction; 2) assess to what extent we could match totally or partially this list to the list of basic emotions defined by Ekman, which are typically featured in visual-based discrete emotion recognition datasets. This pilot, as well as the posterior results of the annotation procedures, defined the final labels to be considered for audio and video channels, which are presented next. Note that, while we tried to consider a comprehensive list of labels, such list may not encompass every emotional expression encountered in real-world deployments, especially across diverse cultures. Therefore, emotionally intelligent HMI systems should include mechanisms for detecting new, unanticipated emotions and support continuous online learning to adapt to these variations.

### C. Annotation protocol

Few works are found in the literature aimed at establishing the amount of emotional information provided through the different audio and video channels. In particular, the study of such channels separately and their combination concludes that the latter does not always yield the best results from both perceptual [79] and computational [22] perspectives, as might be otherwise expected. Previous studies have established that the emotional information provided by each channel or combination strongly depends on the specific emotion, context, and language [80]. For instance, there is vast evidence in the literature that, with respect to dimensional models, arousal can be better detected from the audio channel, while valence is much better estimated from the video channel [81]. With respect to categorical models, *happiness* appears to be more effectively transmitted by video alone, and *surprise* by audio-video [79]. Previous literature has demonstrated that single-channel annotations allow the annotator to avoid increasing the cognitive load and diverging their attention from the perceived emotion [79]. Even when annotating perceived emotional expressions from audio and video simultaneously, humans selectively attend to emotional cues that align closely with their personal and cultural experiences, thus minimizing the cognitive effort required. Consequently, annotators tend to explore the most familiar channel: if rater and rated person are culturally and/or language akin, the rater tends to attend to the auditory signal, whereas when they are culturally distant, they tend to rely more on visual cues [82].

One of the salient attributes considered in the EMPATHIC project is culture and cultural differences, so it was important that the annotation be carried out separately per country by native speakers to be able to capture subtle culture-specific

emotional cues. In addition, the corpus alternates periods of silence, where emotional expressions can only be perceived through video, with speaking bouts, where expressions can be perceived from both channels. Therefore, in order to avoid the annotators' reliance towards a single channel, we decided to separate channels at the annotation level, having different annotators for each channel. This, in turn, results in a richer variety of emotional information from different perceptual channels. Having separate annotations also ensures that the labeled emotional expression can be encountered in the target channel when performing unimodal processing, thus reducing noise for data-driven model learning. Indeed, the importance of channel-specific annotations is increasingly recognized for their ability to improve accuracy, model training, error analysis, and flexibility in real-world applications [83], [84]. Furthermore, they can provide better insights into more complex emotions when there are inconsistencies between audio and facial expressions, e.g. sarcasm. While we focus on channel-specific annotations, future work could incorporate audio-video annotations supported by research on better perceptual annotation methods, to enrich analyses and comparisons.

We employed instructed annotators to be able to control the whole procedure and update it if necessary. Preliminary trials showed that annotators preferred having access to the entire video or speech file instead of annotating isolated snippets due to the presence of context, which helped them make more accurate estimations of the users' emotional state. The annotation process consisted in determining the start and end times of all events associated to given emotions categories throughout a WoZ interaction. To ensure a high inter-rater agreement, we employed a sequential annotation process. Initially, each annotator received a set of files to annotate independently. Subsequently, the within-country inter-rater agreement was calculated with an ad-hoc measure based on event overlap. If the agreement score fell below a predefined threshold, annotators engaged in discussions and re-annotated the files. When the threshold was met, the annotators received the remaining files and continued the process of discussing and re-annotating until the desired level of agreement was attained.

1) *Audio annotations:* These were carried out by listening to the audio signal in Transcriber<fnref n="3" /> with nine native annotators (three per country). For the specific case of audio, the perceived emotions were labeled in terms of both, the categorical and the VAD models. The categorical labels were: *calm/tired/bored, pleased/amused, puzzled, sad, and tense*. The first two labels consist of a combination of similar categories, which was decided after the first annotation rounds as they were highly confused among annotators. For simplicity, we henceforth refer to them as *calm* and *pleased*. Parts of the audio signal with no annotated label are not categorized. The labels assigned to the dimensional VAD model were also discretized for simplicity, and defined as: 1) *positive, neutral, and negative, for valence*; 2) *excited, slightly excited, and neutral, for arousal*; and 3) *dominant, neither dominant nor dominated, and defensive, for dominance*.

The inter-annotator agreement for the categorical annotation

<footnote n="3">³https://transcriber.en.softonic.com/</footnote>
<figure><img src="image_3.png" /><figcaption>Fig. 2: Representation of the segmentation of annotated emotion expression categories to create the gold standard for the audio modality. *Happy* corresponds to *pleased/amused*.</figcaption></figure>

was computed with Cohen's Kappa for each pair of annotators at the millisecond level. SP and NO scored an average coefficient of 0.792 and 0.692, respectively, which indicate substantial agreement [85], while FR scored 0.554, indicating moderate agreement. Once the entire corpus was labeled, we combined the raters annotations using 3-s segments with a 1-s stride (see Fig. 2). To assign an emotion to each segment (i.e., the gold standard), the majority emotion was assigned if that emotion spanned at least 60% of the whole segment. Otherwise, the segment was left without annotation, referred to as discarded. Thus, each segment has four different annotations: one categorical and three for VAD.

2) Video annotations: The annotation was carried out with an in-house software by six native annotators, two from each country. Annotators were instructed to watch muted videos, taking into account only facial expressions and head movements, thus disregarding out-of-face information such as body or hand movements. In addition, they were instructed to watch a short snippet of each user's video (up to 1 min) to familiarize with the user's baseline facial expression. For video annotations specifically, a cross-country calibration was performed after the first set of files was annotated for a small, random subset of videos. This was performed to ensure a common understanding of the instructions, and of the minimum intensity an expression should have to be categorized as such, which can be more objectively determined across countries than for audio-based annotations.

The categorical labels considered were: *sad*, *annoyed/angry* (henceforth referred to as *angry*), *surprised*, *happy/amused* (referred to as *happy*), *pensive*, and *other*. The first four are Ekman's basic emotions [25]. *Pensive* is a mental state rather than an emotional expression; however, it was included in our model as it was found to be a frequent facial expression when users were preparing their response, as in previous HMI-oriented works [57], [86]. Similarly to audio-based annotations, some categories were combined into a single label due to being often confused by annotators. Raters were instructed to annotate as one of the first five categories those segments in which it was clear to them that the expression was present. The label *other* was used to denote those segments in which an expression was occurring but was not included in our expression list, or when more than one expression from the list was present. Finally, all non-labeled instances were considered

<table><caption>TABLE I: Number of audio segments extracted from speech emotional annotations of the EMPATHIC WoZ Corpus.</caption><thead><tr><th></th><th>Calm</th><th>Pleased</th><th>Puzzled</th><th>Sad</th><th>Tense</th><th>Discarded</th><th>Silence</th></tr></thead><tbody><tr><th>Spain</th><td>38359</td><td>833</td><td>1022</td><td>151</td><td>81</td><td>4607</td><td>37910</td></tr><tr><th>France</th><td>19875</td><td>445</td><td>453</td><td>1</td><td>11</td><td>2819</td><td>15978</td></tr><tr><th>Norway</th><td>13960</td><td>474</td><td>44</td><td>0</td><td>0</td><td>1775</td><td>15764</td></tr></tbody></table>

<table><caption>TABLE II: Number of frames extracted from the video emotional annotations of the EMPATHIC WoZ Corpus, corresponding to spoken (top) and silence instances (bottom).</caption><thead><tr><th></th><th>Neutral</th><th>Happy</th><th>Pensive</th><th>Surprise</th><th>Angry</th><th>Sad</th><th>Other</th></tr></thead><tbody><tr><th>Spain</th><td>864112<br>824162</td><td>8163<br>3028</td><td>186735<br>13427</td><td>115<br>56</td><td>0<br>0</td><td>0<br>0</td><td>0<br>0</td></tr><tr><th>France</th><td>484061<br>421980</td><td>28118<br>14385</td><td>98646<br>12915</td><td>162<br>107</td><td>103<br>28</td><td>0<br>0</td><td>693<br>278</td></tr><tr><th>Norway</th><td>345876<br>317166</td><td>11945<br>6253</td><td>67859<br>17303</td><td>72<br>68</td><td>0<br>0</td><td>0<br>0</td><td>239<br>415</td></tr></tbody></table>

to be a *neutral* expression, denoting the baseline face as well as calmed, quiet, or very subtle emotional expressions which do not exceed the consensual expression thresholds.

Post-hoc inter-rater reliability was computed at frame level by means of Cohen's kappa coefficient, achieving a value of 0.7 for SP and FR, and 0.68 for NO, which indicates substantial agreement [85]. We used the intersection between the two annotators to create the final gold standard. Frames with no intersection were discarded for automatic processing, representing around 8% of the total amount of frames.

## D. Analysis of labels

A thorough analysis of corpus annotations is reported in [87]<fnref n="4" />. Here, we summarize the findings, with an emphasis on the categorical labels that will be used in our evaluation.

The number of final audio segments per emotion category is detailed in Table I. As can be seen, *calm* is the most frequent emotion with around 95% of the samples, with respect to instances where the user is speaking and disregarding *discarded*, whereas *sad* and *tense* are quasi absent. Specifically for NO, users rarely showed a *puzzled* expression. With regards to the VAD model, we highlight the following differences: 1) around 30% of FR segments and only 3-4% of SP and NO segments are marked with *slightly excited* for the arousal dimension, while the rest is *neutral*; 2) SP segments are mostly divided between *positive* and *neutral* valence; 3) about 25% of FR segments have *positive* valence, while for NO they are mainly *neutral*; and 4) participants in the three datasets are often *neither dominant nor dominated*.

Table II provides the distribution of emotion categories from video corresponding to spoken (top) and silence (bottom) instances separately. The reported quantities do not include the 0.3% of frames that are not matched to any audio segment, which mainly happened at the end of the video due to audio-video length mismatch. Similarly to audio annotations, video

<footnote n="4">⁴The analysis of [87] was performed with fewer subjects. Nonetheless, its findings are transferable to the sample size used in this paper.</footnote>
<table><caption>TABLE III: Contingency table for audio-video labels. Percentage computed over the total of rows and columns per country, using the audio segments as the unit of measure. See suppl. material (Table I) for a complete version with all the labels.</caption><thead><tr><th rowspan="2"></th><th colspan="3">Spain</th><th colspan="3">France</th><th colspan="3">Norway</th></tr><tr><th>Neutral</th><th>Happy</th><th>Pensive</th><th>Neutral</th><th>Happy</th><th>Pensive</th><th>Neutral</th><th>Happy</th><th>Pensive</th></tr></thead><tbody><tr><th>Calm</th><td>78.06</td><td>0.34</td><td>17.08</td><td>76.18</td><td>3.25</td><td>16.19</td><td>78.60</td><td>1.27</td><td>16.63</td></tr><tr><th>Pleased</th><td>1.34</td><td>0.36</td><td>0.18</td><td>1.21</td><td>0.87</td><td>0.06</td><td>2.25</td><td>0.84</td><td>0.12</td></tr><tr><th>Puzzled</th><td>2.07</td><td>0.004</td><td>0.57</td><td>1.65</td><td>0.02</td><td>0.58</td><td>0.21</td><td>0</td><td>0.08</td></tr></tbody></table>

annotations lead to highly imbalanced results. *Pensive* was the most frequent manually labeled expression, appearing 11% of the time, followed by *happy*, present in 2% of the total images. Despite these findings, the *neutral* category clearly dominates over all categories, appearing around 87% of the time.

As observed, the main challenges encountered in the EM-PATHIC WoZ corpus are: 1) the imbalance between the different emotion classes; and 2) the imbalanced number of subjects across countries and limited data samples, particularly for audio. The former indicates that the interaction with the VC did not lead the users to experience strong emotions like *sad*, *angry* and *surprise*. This is in line to what is usually observed in real, spontaneous HMI interactions, and aligns with recent evidence showing that older adults express significantly more positive emotions than negative ones [88]. In addition, many of the users may have an *a priory* positive attitude since they are volunteers to participate in the experiment. The reduced number of audio samples is partly caused by the amount of time that participants had to wait for the WoZ to respond.

The high class imbalance poses a challenge for data-driven models to effectively learn discriminative information for minority classes. To address this, a deployed system would require data augmentation or imbalance handling mechanisms to leverage all the classes, as well as benefit from potential complementary datasets. Nonetheless, for this study, we focus on the three most represented categories for each label type, so as to minimize confounding effects from sample size disparities and have a sufficient sample size for a meaningful analysis on the corpus, even though differences in sample sizes remain. That is, we maintain *calm*, *pleased*, and *puzzled* for audio, and *neutral*, *happy*, and *pensive* for video.

Table III depicts the relationship among audio-video labels per country, using the audio segments as reference and computing the most repeated video category for the valid frames within the start-end times of an audio segment. We find that audio-based *calm* and video-based *neutral* coincide 76-79% of the time (66-70% if including all labels). However, there is no evident one-to-one correspondence for the remaining cases. Given that each channel contributes distinct information, we retain the two label types as independent entities, allowing the system to estimate both of them at each time step.

## IV. METHODOLOGY

In this section, we describe our methodology and training strategy for data-driven emotion expression recognition using different modalities. The methodological choices depend on the EM-PATHIC-VC system requirements, following those of common multi-agent systems [1]. The *human sensing* module,

which includes emotion recognition, is one of the multiple modules that must communicate timely with a *dialogue manager*. The latter controls the conversation flow by integrating the information from *human sensing* and other modules to transfer the appropriate VC reactions to the *natural language generation* and avatar animation modules [3], [24]. In case of conflicting per-channel estimates, the VC reactions could be adapted to progressively gather more signal about the user's state. In the final system, some modules would be located in remote servers and data transfers would be done via network. Thus, efficiency in the whole process is crucial to ensure a seamless and natural interaction. Therefore, we prioritize independent, lightweight submodules for each channel, which can operate asynchronously and produce estimates at the lowest granularity level for further processing.

Fig. 3 shows an overview of our methodological pipeline. In summary, we first extract features from the different modalities. More specifically, for the main modalities (i.e., speech from audio and facial expressions from video), we train individual models for their respective labels on all the available data to learn rich emotional features. In parallel, we extract additional features from video, namely *looking-at-VC*, head, 3D gaze, and eye movement information. Since the features of each modality are extracted at different time resolutions (i.e., audio features every 3 s, facial features at every frame, and additional features every 1.5 s), we apply fixed modality synchronization tailored to each label type, which allows us to perform the cross-channel and multimodal evaluation. Finally, the previously extracted features are combined and further evolved with an MLP to recognize the user's emotional state for the audio- and video-based labels separately.

### A. Speech features from audio

In this work, we only consider those audio segments with an associated emotion while the user is speaking and for which the avatar speaks less than one-third of the segment duration.

First, we choose the WavLM speech model [39] to extract the acoustic information of each segment using the raw signal waveform. Due to the limited audio sample size, incorporating a large-scale pretrained speech model allows us to increase generalization ability. In particular, WavLM was trained on 94k hours of English-spoken audios extracted from three large-scale speech datasets, and it has been shown to obtain high performance in SER, among other tasks. Despite the language mismatch, our previous studies have demonstrated WavLM's ability to outperform other approaches on the EM-PATHIC WoZ Corpus without fine-tuning, including large pretrained speech models and models trained on handcrafted
<figure><img src="image_4.png" /><figcaption>Fig. 3: Methodological pipeline overview, following the information flow through different EMPATHIC-VC submodules.</figcaption></figure>

features [32]. We extract the features of its last hidden states, which outputs a 1024D vector every 1/50 s. This results in 150 feature vector instances per segment. We compute the average along the time dimension to reduce the final length to 1024.

Then, we feed such features to four two-layer MLPs: one for categorical emotional state recognition and three for the dimensional model. Although the main goal is to perform categorical recognition, we decided to include the dimensional model to leverage the available annotations and enrich the feature representation, given the relationship between the two models [29]. We also chose to train them separately since each task may converge at a different rate; thus, a multi-task learning approach would not be optimal for all the outputs. The first layer reduces the 1024 extracted features to 64 with ReLU as the activation function, while the second one is in charge of extracting the logits for the prediction of the emotional states via softmax. Cross-entropy is used as a loss function, and the Adam optimizer is used to train the four networks with a learning rate of 0.001 over 5K iterations. To deal with the imbalance of the data, the sampling probability for the samples of the minority classes is four times higher than calm.

Finally, the logits of the four models are concatenated to the computed WavLM features in a hybrid fusion fashion, resulting in a 1034D feature vector<fnref n="5" />. This way, we preserve the generic speech representation and augment it with a reduced set of domain-specific information. We refer to this feature set, and consequently to this modality, as A.

## B. Facial expression features from video

For this work, we adopt a static-based approach, despite dynamic ones usually being better suited for this task [11]. In our context, issues such as frame loss during data transfer or limitations in network capacity leading to variable frame rates may impede the effective utilization of fine-grained dynamics.

For each video frame, we first detect faces using Face-Boxes [89] and estimate 68 facial landmarks in the image space by means of 3DDFA_v2 [90]. Less than 1% of the data is lost on these steps. Using these landmarks, the face is rotated, scaled, and cropped to obtain a normalized RGB image of 224x224 pixels. Then, we use the Xception CNN model [91] pretrained on ImageNet [92] to extract discriminative features from the face images, and add four fully connected dense layers to the top of the network, each followed by ReLU and dropout (with a rate of 0.5), in addition

to a final softmax layer for FER. During optimization, we found that the best strategy was to freeze the first 70 layers of Xception and fine tuning the last 10. Consequently, we finetune such layers and train the added ones from scratch on both spoken and silent instances. According to this transfer learning scheme, we get a total of 23.6M parameters, where 16.5M are trainable, and the remaining 7M are non-trainable. Training is based on the Adam optimizer, with a learning rate of 0.001. To tackle the class imbalance issue, we use a weighted cross-entropy loss function where the weight of each emotion class is associated with the inverse frequency in the training set.

Lastly, we extract the output 256D features from the last hidden layer. We refer to this feature set and modality as F.

## C. Additional features from video

We also use the video stream to compute a series of additional features based on the per-frame estimated 3D gaze direction vector (represented as horizontal and vertical gaze angles) and head pose (yaw, pitch, and roll).

To do so, we first fit a 3D face morphable model [93] to the detected 2D landmarks from Sec. IV-B and apply perspective-point [94] to estimate the 3D head position and orientation, as well as the 3D eye position as the gaze origin. A normalized face image is fed to the 3D gaze estimation model ETH-XGAZE, trained on a dataset of homonymous name [95]. Although none of the existing gaze estimation datasets include older adults, a qualitative examination of the estimated gaze direction using this model showed that performance was mostly impacted for users wearing colored lens or eyeglasses with substantial reflection caused by the computer screen. We did not detect blinks or pupil size due to their low reliability in our scenario. To reduce noise, the estimated head pose and gaze trajectories are postprocessed with a 5-frame median filter. Combining head pose $\mathbf{h} = (y, p, r)$ and gaze $\mathbf{g} = (x_g, y_g)$ vectors, we further convert the 3D gaze direction into an eye-in-head gaze direction vector, that is, mimicking eye rotation $\mathbf{e} = (x_e, y_e)$. The three data sources are filtered to discard invalid data (e.g., frames with incorrectly detected faces, or for which head or eye movements are not anatomically plausible). Following other works on eye and head pose processing [63], [96], all trajectories are processed using a sliding window of 1.5 s and stride of 1, and centered at every half second throughout the video. Windows smaller than 0.5 s or for which more than 50% of the frames are invalid are discarded, accounting for around 2% of the windows. We compute a 227D feature vector for each window, containing information from the three data sources represented as functionals of the trajectories (see Table IV), and a complementary attention measure, described below. Due to the effect of glasses on the resulting eye trajectories, we add a manually annotated ternary flag to denote whether the participant is wearing glasses, and if so, whether the eyes are clearly visible. We refer to the resulting 228D feature set and modality as G.

1) *Looking at VC:* The EMPATHIC-VC system can use this measure to estimate whether the user is engaged with the VC. Here, we largely follow [97] to estimate the location of the VC, wherein it is assumed that the zone (cluster) in the

<footnote n="5">⁵We use two categories for valence and arousal instead of the original three (see Sec.III-C1) due to the low number of samples.</footnote>
<table><caption>TABLE IV: Functionals computed for each element of the additional modalities (3D gaze vector, eye rotation, and head pose). x., y.: horizontal and vertical 3D gaze/eye rotation components, respectively; y: head yaw; p: pitch; r: roll.</caption><thead><tr><th>Element</th><th>Functionals</th></tr></thead><tbody><tr><td>x., y.,<br>abs Δx., abs Δy.,<br>abs Δx./t, abs Δy./t,<br>abs Δg/t, abs Δe/t,<br>y, p, r,<br>abs Δy., abs Δp., abs Δr</td><td>min, max, mean, SD,<br>range (except for abs Δx. and abs Δy.,<br>25th perc, 50th perc, 75th perc, IQR</td></tr><tr><td>Δx., Δy., Δg., Δe,<br>Δy., Δp., Δr,<br>abs Δy/t, abs Δp/t, abs Δr/t</td><td>mean, SD</td></tr></tbody></table>

SD: Standard deviation; perc: Percentile; IQR: interquartile range.

3D space with the highest density of gaze vectors intersecting the camera plane is where the VC is located. We create a 6D one-hot encoding vector denoting the looking-at-VC likelihood from lower to higher, based on the distance from the gaze point to the cluster. Per-valid-frame vectors are averaged over a time window, producing a 6D vector per window.

2) **3D gaze direction:** We compute g functionals of: per-component (i.e., $x_g, y_g$) gaze angles, per-component angle differences (e.g., $\Delta x_g$) and their magnitude (e.g., $\Delta x_g$) between any two consecutive frames, direction ($\Delta g$) and speed ($\Delta g/t$) of the gaze vector between any two consecutive frames, and per-component speed (e.g., $\Delta x_g/t$) between any two consecutive frames. This results in a 67D feature vector.

3) **Eye rotation:** We compute the same functionals for e as for g, resulting in a 67D feature vector.

4) **Head rotation:** Lastly, we compute h functionals for the following: per-component (i.e., $y, p, r$) head pose angle, per-component angle differences (e.g., $\Delta y$) and their magnitude (e.g., $\Delta y$) between any two consecutive frames, and per-component speed (e.g., $\Delta y/t$) with respect to any two consecutive frames. This results in a 87D feature vector.

### D. Temporal synchronization of modalities

In order to effectively integrate and analyze the multimodal data captured from different sources, we employ a fixed modality synchronization approach per label type.

For the audio-based evaluation, for which the system outputs an estimate every 3 s, we compute the average and SD of the available per-frame F features within an audio segment, resulting in a 512D vector. This provides a robust facial expression descriptor that is less susceptible to accidental fluctuations despite disregarding facial temporal dynamics. Preliminary experiments evaluated a second version, consisting in concatenating the features of the most central frame of each second of the audio segment, hence maintaining such dynamics. However, the former version outperformed the latter for the majority of settings. Regarding G, we use the window aligned to the center of the audio segment, thus discarding those windows at the extremes of the segment.

Conversely, for video-based evaluation, the temporal resolution is increased to frame level. Thus, each G window and A segment are used multiple times and matched to different

frames. In particular, we associate each frame with a specific G window and A segment based on its closest proximity to the central timestamp of the respective window and segment.

This way, all F frames, G windows, and A segments receive audio and video labels. For each evaluation case, feature sets that do not have correspondence due to missing data of any of the modalities are omitted, resulting in around 86% of the original data for audio and 98% for video.

### E. Final models

The extracted features from a given modality are normalized according to the training set range and fed to a 2-layer MLP with ReLU activation and dropout of 0.5, followed by a softmax dense layer for classification of a given label type. We evaluate three low-complexity MLP configurations, with number of hidden units 100-20, 200-40, and 500-100. For multimodal evaluation, the feature sets of the different modalities are concatenated before being fed to the MLP. We evaluated other attention-based fusion approaches in preliminary experiments, such as self- and cross-modal attention [98]. However, their performance was equivalent to concatenation, so we proceed with the latter for the experimental evaluation.

We tackle data imbalance by randomly sampling instances of each class with the same probability. Additionally, due to the small sample size of the audio-based evaluation, we employ an oversampling strategy such that each sample of the minority class (pleased for SP and FR, and puzzled for NO and WH) is utilized around three times per epoch. To maintain an approximate balance between classes, the other classes are sampled a similar number of times. The training samples per epoch are thus set to 5418 samples for SP, 2556 for FR, 234 for NO, and 10494 for WH. Conversely, since the sample size for video-based evaluation is considerably higher but also contains higher redundancy, we set the training sample size to 7500 for all countries. Samples are randomly selected; thus, at the end of the training stage, all samples from the minority classes are seen multiple times, while for neutral, only a fraction is seen.

All evaluated models are trained with cross-entropy loss, Adam optimizer, learning rate of 0.0001, and batch size of 64. We empirically set the number of training epochs to 100 for all countries and evaluations except for NO with audio-based labels, for which we train for 200 epochs.

## V. EXPERIMENTAL EVALUATION

### A. Research questions

The characteristics of the EMPATHIC WoZ scenario allow us to evaluate the contribution of the different modalities for the considered emotions in various contexts. First, we separately consider the evaluation scenario with audio-based labels and the one with video-based labels. We have a main modality for each label type: A for audio-based and F for video-based labels. We refer to the remaining modalities (e.g., F and G for audio-based evaluation) as auxiliaries for that evaluation scenario. Main and auxiliary modalities can be combined to improve performance. Each evaluation is performed in each country individually (SP, FR, and NO) and on WH. The latter allows us to evaluate trends of the complete set of data and
quantify the effect of training with country-specific data in comparison to a larger multi-country set. The audio-based scenario only includes data where the user is speaking. By contrast, for the video-based scenario, we can compare the performance of evaluating spoken content to silent content. What is more, as for the country-oriented evaluation, we can assess the effect of training the final video-based model with speaking-status-specific data in comparison to with all data.

On this basis, we aim to answer the following eight research questions: $RQ_1$) Can the main modality for a given label type obtain the same discriminative power for all the classes considered?; $RQ_2$) Can the auxiliary modalities achieve similar performance to the main modality?; $RQ_3$) Is multimodality beneficial?; $RQ_4$) Are there noteworthy differences in performance among countries?; $RQ_5$) Does training with data from multiple countries prove beneficial with respect to country-specific training?; $RQ_6$) For video-based evaluation, does training with spoken and silent instances prove beneficial with respect to spoken/silent-specific training?; $RQ_7$) Are there any performance differences between spoken and silence instances?; and $RQ_8$) Are there any performance differences between audio- and video-based evaluation?

## B. Evaluation protocol

We build 10-fold subject-independent training and test splits for each country subset (SP, FR, NO) and a fourth one with the data of all the countries (WH) following approximately a 9:1 ratio. The folds for WH contain the same subjects as the per-country folds. Architecture selection (i.e., the number of MLP hidden units) and hyperparameter tuning are carried out independently per experiment based on random validation subpartitions of the training splits. For each experiment, the best configuration over all folds is then retrained on the whole per-fold training split and used for all folds. Best architectures per experiment are reported in Sec. B-D of the supplementary material. We perform 10-fold cross-validation three times following the same splits for all models to account for the stochasticity of the data sampling and whole learning process.

Fold-wise performance is measured using the unweighted average accuracy (i.e., unweighted average recall), which gives the same weight to the accuracy of each class regardless of the number of samples for each class. Per-class accuracy is thus equivalent to per-class recall (i.e., the number of samples predicted correctly out of the total number of samples for a given class). Note that the test splits of some folds do not contain all classes, especially *puzzled* for NO. In such cases, the average accuracy is computed only for the classes that have at least one test sample. We also perform multiple pairwise comparisons with the corrected repeated k-fold cross-validation t-test [99] to test for statistically significant differences (p<.05) among average accuracy results. We control for false discovery rate with the BKY correction [100], grouped by country subset.

Due to the large amount of experiments, in the following subsections we report the results for the WH dataset, trends observed across different countries, and noteworthy highlights specific to each country. For additional results on a per-country basis, please refer to Sec. C, D and E of the supplementary

<table><caption>TABLE V: **Audio-based results on WHOLE**, reported as unweighted avg. accuracy ± SEM over 10 folds and 3 runs per fold. Bold: best acc. per group. Underlined: best acc. overall.</caption><thead><tr><th>Modality</th><th>Calm Accuracy</th><th>Pleased Accuracy</th><th>Puzzled Accuracy</th><th>Average Accuracy</th></tr></thead><tbody><tr><td>A</td><td>76.42 ± 0.8</td><td>63.54 ± 1.3</td><td>59.89 ± 2.3</td><td>66.61 ± 0.6</td></tr><tr><td>F</td><td>15.9 ± 1.3</td><td><strong>69.91 ± 3.2</strong></td><td><strong>63.41 ± 3.0</strong></td><td><strong>49.74 ± 0.9</strong></td></tr><tr><td>G</td><td><strong>25.26 ± 1.6</strong></td><td>49.74 ± 2.3</td><td>44.66 ± 2.0</td><td>39.89 ± 0.9</td></tr><tr><td>A+F</td><td>75.68 ± 0.5</td><td><strong>67.59 ± 1.6</strong></td><td><strong>61.65 ± 2.2</strong></td><td><strong>68.31 ± 0.5</strong></td></tr><tr><td>A+G</td><td>76.62 ± 0.7</td><td>62.38 ± 1.4</td><td>58.52 ± 2.3</td><td>65.84 ± 0.6</td></tr><tr><td>A+F+G</td><td><strong>76.93 ± 0.7</strong></td><td>67.11 ± 1.7</td><td>60.41 ± 2.7</td><td>68.15 ± 0.6</td></tr></tbody></table>

material. Country sets used for training and testing are denoted as training country→testing country (e.g., WH→SP to denote training with WH and testing with SP). The WH models selected for the country-specific comparison are the ones that worked better for the WH validation sets, so the reported performance would be different if selecting the best models for each country independently. Likewise, models trained on WH with silence and speech data are also the ones that worked better for the WH validation sets.

## C. Audio-based emotion expression recognition results

Table V summarizes the results for the different experiments with audio-based labels on the WH dataset. We report below the results with respect to each research question.

1) Main modality: A obtains higher accuracy with calm followed by pleased, correlated with the number of samples per class. Puzzled gets more confused with calm than pleased. In general, results are more stable across runs than across folds: the mean SD across folds is 3.3%, while the mean SD across runs is 0.8%. Thus, changes in the standard error of the mean (SEM) generally denote higher variability across folds.

2) Auxiliary modalities: Despite the high accuracy obtained by F for *pleased* and *puzzled*, confusion patterns reveal that *calm* is mostly confused with *puzzled*, which does not occur for *pleased*. This implies that F provides information that is particularly discriminative and potentially correlated to the audio modality specifically for the latter class. By further analyzing the F predictions and their correlation to facial expression categories, we observe that *pleased* is detected for 97% of the audio segments for which the majority facial expression is *happy*, despite only 30% of them having the *pleased* annotation. Thus, the same features that correspond to a *happy* face are related to the facial features corresponding to a *pleased* speech. Instead, G's results are slightly over random performance, indicating that G alone is not informative enough to recognize the classes considered. All unimodal (A, F, G) and unimodal vs bimodal (F vs A+F, G vs A+G) pairwise comparisons are significantly different (p<.0001).

3) Multimodality: Overall, incorporating F improves performance over A alone. By contrast, incorporating G seems detrimental on average. The best multimodal approach for WH, A+F, achieves a 2.5% relative performance increase over A. Class-wise, we observe that adding G increases performance and stability for calm, while adding F is beneficial
<figure><img src="image_5.png" /><figcaption>Fig. 4: **Per-country audio-based results**, training on either SP, FR, NO, or WH training sets, and evaluating on SP, FR and NO test sets. Reported as unweighted avg. accuracy ± SEM over 10 folds and 3 runs per fold.</figcaption></figure>

for *pleased* and *puzzled*, despite the stability of the latter decreases. Statistical tests further confirm that A+F vs A+G (p=.038) and A+G vs A+F+G (p=.024) differ significantly. SP and NO follow similar trends class-wise, although for them, G is also beneficial for *puzzled* but to a lesser extent than A. For FR we observe an inverse trend, which we comment below.

4) *Comparison across countries:* Fig. 4 depicts per-country average accuracy results. With respect to multimodality, for SP and NO, we observe a similar trend to that of Table V, with A+F and A+F+G outperforming A alone. For FR, however, accuracy decreases as the number of features increases, obtaining the highest average accuracy overall with A (66.28%), which evidences variations among countries. First, FR obtained the lowest inter-agreement score (Sec. III-C1). Analyzing the FR dataset distribution, we find that the average SD of the FR audio features is slightly larger than that of the other countries (0.68 for FR, 0.65 for SP, and 0.63 for NO), indicating that the data is more dispersed in the feature space. Furthermore, the best models for FR are the ones with the lowest number of parameters, as opposed to NO, which requires the largest number of parameters despite having a similar sample size (Sec. B-D of the supplementary). These results indicate that adding more features to FR increases the risk of overfitting, thus decreasing performance. Class-wise, in contrast to WH results, both G and F aid in *puzzled* recognition for SP, while both aid in *calm* recognition for FR. Additionally, FR obtains the highest accuracy for *pleased* with A alone, while for the rest of the classes and countries, multimodal models outperform A. With respect to the auxiliary modalities, NO and FR appear to leverage F and G better than SP, respectively. Per-class accuracies are redistributed with respect to WH.

5) *Expanding training data including other countries:* Fig. 4 also depicts the effect of training with the WH dataset instead of each country separately. As can be seen, adding

<table><caption>TABLE VI: Video-based results under speech on WHOLE, reported as unweighted avg. accuracy ± SEM over 10 folds and 3 runs per fold. Bold: best accuracy per group. Underlined: best accuracy per training type. Italics: best accuracy overall.</caption><thead><tr><th>Modality</th><th>Neutral Accuracy</th><th>Happy Accuracy</th><th>Pensive Accuracy</th><th>Average Accuracy</th></tr></thead><tbody><tr><td colspan="5">Training on speech data:</td></tr><tr><td>F</td><td>73.98 ± 1.0</td><td>72.2 ± 2.4</td><td>57.87 ± 2.1</td><td>68.02 ± 1.1</td></tr><tr><td>A</td><td>39.07 ± 1.1</td><td><b>63.57</b> ± <b>2.1</b></td><td>66.67 ± 1.5</td><td>56.44 ± 0.6</td></tr><tr><td>G</td><td><b>51.13</b> ± 1.8</td><td>46.33 ± 2.4</td><td><b>74.47</b> ± <b>2.2</b></td><td><b>57.31</b> ± <b>0.5</b></td></tr><tr><td>F+A</td><td>74.5 ± 0.9</td><td><b>73.39</b> ± <b>2.4</b></td><td>60.95 ± 1.9</td><td>69.61 ± 1.1</td></tr><tr><td>F+G</td><td><b>76.45</b> ± <b>1.1</b></td><td>71.58 ± 2.4</td><td>66.86 ± 2.2</td><td>71.63 ± 1.1</td></tr><tr><td>F+A+G</td><td>76.36 ± 1.0</td><td>73.27 ± 2.4</td><td><b>68.52</b> ± <b>1.9</b></td><td><b>72.72</b> ± <b>1.0</b></td></tr><tr><td colspan="5">Training on all data (speech + silence):</td></tr><tr><td>F</td><td>70.44 ± 1.1</td><td><u>73.44</u> ± <u>2.2</u></td><td>61.58 ± 2.1</td><td>68.49 ± 1.1</td></tr><tr><td>G</td><td>42.45 ± 2.0</td><td>55.56 ± 2.1</td><td><u>76.96</u> ± <u>2.0</u></td><td>58.32 ± 0.5</td></tr><tr><td>F+G</td><td><u>73.55</u> ± <u>1.1</u></td><td>73.11 ± 2.2</td><td>70.56 ± 2.1</td><td><u>72.41</u> ± <u>1.0</u></td></tr></tbody></table>

more data from different countries consistently improves accuracy on average for SP and NO with A-based models, although stability decreases. NO trained on WH obtains the highest accuracy overall (70.13% with A+F). For FR, however, we find the opposite effect again, with a decrease in accuracy of up to 4.1% for A, where *pleased* is the most affected class. By contrast, the behavior of the auxiliary modalities is the opposite, with F obtaining the highest accuracy for this class, which additionally increases with respect to FR→FR. Considering previous findings, we hypothesize that the FR audio feature distribution of the *pleased* class is significantly different than that of SP and NO; thus, adding more data is detrimental. Another difference comes from the arousal distribution of FR, being the country with the highest number of annotated *excited* instances (61.75% compared to 12-18% for the other countries). Continuing with class-wise results, SP obtains performance increases for all classes in a similar proportion, benefiting from the increased data variability and sample size. By contrast, *calm* performance decreases for NO, which may be caused by the significant increase in the number of training instances for the minority classes (around 279% and 4369% increase, respectively). With respect to auxiliary modalities, the general trend shows that adding more data hurts performance. By further analyzing confusion patterns, we observe that these are mostly reversed when training with WH, especially affecting discrimination between *calm* and the other classes for G and *calm* and *puzzled* for F. For the latter modality, though, *pleased* is still recognized accurately.

## D. Video-based emotion expression recognition under speech

Table VI summarizes results for video-based labels under speech trained and evaluated on WH, using two different training regimes: 1) training on samples where the user is speaking (*speech data*); and 2) training on all samples irrespective of speaking status (*speech+silence*). We report the results below.

1) *Main modality:* F alone obtains similar accuracy for *neutral* and *happy*, while comparatively struggles with *pensive*, which is highly confused with *neutral*. This behavior is not
<figure><img src="image_6.png" /><figcaption>Fig. 5: **Per-country video-based results under (a) speech or (b) silence**, trained on SP, FR, NO, and WH training sets, and evaluated on SP, FR and NO test sets. Reported as unweighted average accuracy ± SEM over 10 folds and 3 runs per fold.</figcaption></figure>

proportional to the number of instances since *pensive* has more than *happy*. Nonetheless, *happy* performance is slightly less stable than that of *pensive*. Country-wise, however, the performance gap is found between *neutral* and the minority classes. The SD across folds is 6.2%, higher for this scenario than for audio but more consistent. By contrast, the SD across runs is 0.05%, unveiling the large variability across subjects.

2) **Auxiliary modalities:** A and G obtain accuracy results closer to the main modality than for the audio-based scenario, with G slightly outperforming F on average. Statistical tests show significant differences for F vs A/G (p=.015), and F vs F+G/A (p<.001) when training with speech data, and for all comparisons when training with all data (p<.001). Remarkably, G achieves the highest accuracy overall for *pensive*. A is informative for *pensive* as well but to a lesser extent, also outperforming F, and is more informative than G for *happy*. Nonetheless, G is less stable than A class-wise, although on average, they are more stable than F. Analyzing the confusion patterns for all datasets, we confirm that G is highly discriminative for *pensive*, and A for *happy*.

3) **Multimodality:** When training on speech data, we observe that adding A or G to F increases accuracy, and the highest is achieved by combining the three modalities, showing a 6.9% relative improvement over F alone. Class-wise, adding G substantially improves performance for *pensive* followed by *neutral*, while adding F has a more subtle effect. By contrast, *happy* appears to benefit from A and not G, but does so when combining the three modalities. We observe similar trends when training on all data. Statistical tests confirm significant differences for the following cases when training on speech data: F vs F+G (p=.031), F vs F+A+G (p=.018), F+G vs F+A (p=.044), and F+A vs F+A+G (p=.015); and when training on all data: F vs F+G (p=.015). Trends are overall maintained across countries, with some differences highlighted below.

4) **Comparison across countries:** Fig. 5a illustrates per-country average results for all modalities and the two training regimes. In general, SP achieves the highest accuracy for all

modality combinations, greatly benefiting from G followed by A when added to F. It obtains the highest accuracy overall with F+A+G (69.34%) when training with speech, and with F+G (69.33%) when training with all data. NO achieves a similar accuracy with the trimodal model, although it benefits more from A than from G. By contrast, FR barely benefits from adding A, and repeatedly scores the lowest, despite having slightly more data than NO and almost equal performance on average with F. This might be partly caused by the difference in class proportions across countries, with *happy* being the most variant class (4.7% of the total data for FR, 0.8% for SP, and 2.7% for NO). The difference in the distribution of audio features for FR (discussed in Sec. V-C) is also noticeable here, with A alone obtaining the lowest accuracy for FR, and with a substantial difference compared to the other countries. Adding A to F hurts *pensive* recognition for FR due to a high confusion between *neutral* and *pensive*, although their performance alone is better, and the highest with A. With respect to G, we observe the highest discriminative power for *pensive* with SP, with more elevated levels of *neutral-happy* confusion for the other two countries, thus scoring lower in the comparison.

5) **Expanding training data including other countries:** Fig. 5a also illustrates the effect of training with WH instead of each country separately for the two training regimes. In general, adding more data increases accuracy on average for all cases except for A, for which accuracy is mostly maintained or slightly reduced. As can be seen, FR obtains the highest performance increase overall despite still scoring the lowest, and NO obtains the highest accuracy results overall, with F+A+G being the top performer (74.24%). SP and FR also achieve the highest accuracy with F+A+G. Models are slightly less stable when training on WH for F-based models, except for NO. SP obtains the lowest gain, highly likely due to being the country with the most instances in WH for all classes except for *happy*. However, when we investigate class-wise trends, we observe an interesting difference. For NO and FR, the minority classes have their accuracy significantly increased, and the *neutral*
accuracy decreased for all modality combinations, proving that the increase in variability and effective training data is beneficial for them to decrease confusion with *neutral*. By contrast, SP sees the *neutral* accuracy increase with all F-based models for all cases and with G only when training with speech data, while F and F+G maintain accuracy for *pensive* and decrease it for *happy*. However, performance increases when adding A for the minority classes, although A alone gets *happy* accuracy reduced, while with G alone they both improve. For silence-based evaluation (Sec. V-E), *happy* accuracy does increase with F-based models when training on WH. Thus, we believe this difference in *happy* performance is due to the facial deformations caused by speaking being different across countries, greatly increasing variability. Although this is true for all countries, due to the small number of *happy* instances for SP, this distribution might be narrower than that of FR and NO. Therefore, increasing variability could be detrimental. Then, when adding A to F, A helps discriminate better among classes that visually might be more similar.

6) Expanding training data including silence instances: As can be seen in Table VI for WH, training on all data marginally but consistently improves performance on average, with the highest improvement obtained with G. Class-wise, confusion patterns reveal that the minority classes are less predicted as *neutral*, with a slight increase in confusion in the other direction in some cases. Nonetheless, the change is positive for *pensive* and *happy*. For G specifically, the *neutral-happy* confusion patterns are inverted. The consistent improvement is also observed across countries, as depicted in Fig. 5a, both when training per country and when training on WH, and the class-wise trends are generally maintained. As a matter of fact, per-class accuracies tend to be more balanced in this setting. This indicates that, by including training instances with no facial deformations caused by speaking, the models can pick up other cues that are consistent regardless of speaking status, which helps detect more actual *happy* and *pensive* instances. Specifically, *pensive* always obtains the highest accuracy overall, with a slight performance increase when training on WH with models including G. The other classes also increase their accuracy when training with all data, although the highest accuracy is obtained from including A, which can only be accomplished when training with speech instances.

## E. Video-based emotion expression recognition under silence

Table VII summarizes results for video-based labels under silence trained and evaluated on WH, using two different training regimes: 1) training on samples where the user is not speaking (*silence data*); and 2) training on all samples irrespective of speaking status (*silence+speech*).

1) Main modality: For WH, results match the video-under-speech scenario on average and per class, with a decrease in stability. The *neutral-pensive* confusion also decreases. Country-wise, trends are similar to such scenario but differ from WH. For SP, *pensive* is the top performer (73.3% accuracy) followed closely by *neutral*, while for FR and NO, the top performer is *neutral* by a great margin (around 78-80% accuracy). For NO specifically, *pensive* obtains extremely low

<table><caption>TABLE VII: Video-based results under silence on WHOLE, reported as unweighted avg. accuracy ± SEM over 10 folds and 3 runs per fold. Underlined: best accuracy per training type. Italics: best accuracy overall.</caption><thead><tr><th>Modality</th><th>Neutral Accuracy</th><th>Happy Accuracy</th><th>Pensive Accuracy</th><th>Average Accuracy</th></tr></thead><tbody><tr><td colspan="5">Training on silence data:</td></tr><tr><td>F</td><td>73.33 ± 1.8</td><td>74.33 ± 3.2</td><td>57.76 ± 2.8</td><td>68.47 ± 1.5</td></tr><tr><td>G</td><td>61.3 ± 1.6</td><td>48.15 ± 3.4</td><td>74.09 ± 1.2</td><td>61.18 ± 0.8</td></tr><tr><td>F+G</td><td>79.45 ± 1.3</td><td>72.76 ± 3.2</td><td>71.74 ± 1.8</td><td>74.65 ± 1.2</td></tr><tr><td colspan="5">Training on all data (speech + silence):</td></tr><tr><td>F</td><td>79.56 ± 1.4</td><td>72.3 ± 3.4</td><td>51.35 ± 2.8</td><td>67.74 ± 1.5</td></tr><tr><td>G</td><td>67.34 ± 1.4</td><td>43.54 ± 3.4</td><td>72.3 ± 1.5</td><td>61.06 ± 1.1</td></tr><tr><td>F+G</td><td>85.23 ± 1.0</td><td>71.35 ± 3.4</td><td>61.98 ± 1.8</td><td>72.85 ± 1.1</td></tr></tbody></table>

accuracy (38.3%). As usual, the latter is mostly misclassified with *neutral*, but also with *happy* to a lesser extent, a behavior we had not observed until now. Compared to the video-under-speech scenario, the proportion of *neutral* instances under silence is higher (around 93-98% depending on the country), which might explain the accuracy bias. For the other classes, however, accuracy results do not generally match class proportions, with *happy* corresponding only to 1.4% of the WH sample size but scoring very close to *pensive*. The SD across folds is 8.5%, slightly higher than the speech scenario. The SD across runs has a fairly larger range (mean of 0.1%).

2) Auxiliary modality: Following previous trends, G alone achieves the highest accuracy and discriminative power for *pensive*. On average, the accuracy for G gets the closest to F compared to audio and video-under-speech scenarios, with no significant differences between them. Interestingly, for NO, *pensive* is misclassified as *happy* at a similar proportion compared to the main modality.

3) Multimodality: Adding G to F consistently increases accuracy on average, following previous trends class-wise, i.e., increasing accuracy and stability for *neutral* and *pensive*, while not contributing for *happy*, for which F alone is usually the top performer. As a matter of fact, *pensive* is the most benefited class for WH and across countries, and *neutral* obtains the highest accuracy overall across training regimes. No statistically significant differences are found, likely due to the high SEM of F alone, which decreases with multimodality.

4) Comparison across countries: Fig. 5b depicts per-country average accuracy results. As can be seen, all countries benefit from multimodality, with FR benefiting the most when training on silence data. SP achieves the highest accuracy for all settings by a large margin, obtaining the best accuracy overall with F+G (74.35%) when training with silence data. Contrary to the video-under-speech evaluation, NO scores the lowest. With respect to the auxiliary modality G, its difference with respect to F is small for FR for both training regimes and for NO only when training when silence, as the performance of G decreases when training with all data. Class-wise, trends vary compared to WH depending on the modality, as already discussed above. Again, one highlight is the low accuracy and
stability for *pensive* with either modality for NO compared to
the other countries and classes, as well as its confusion with
*happy*. In fact, NO is the country with the highest number of
*pensive* instances (around 5.4% of the NO sample size), which
might indicate a difference in the annotation procedure.

5) *Expanding training data including other countries:* Fig. 5 also shows the effect of training with WH. Similarly to the video-under-speech evaluation scenario, SP barely benefits from adding cross-country data compared to the other countries. Still, the highest accuracy is obtained by SP with F+G (75.4%) when training with silence. For FR, the highest performance increase is obtained by F while, for NO, G is the most benefited modality. Class-wise, we again observe differences between SP and the other two countries. For NO and FR, F-based models increase performance for *happy* and *pensive* and reduce it for *neutral*, while for SP, *neutral* and *happy* performance increases while *pensive* performance decreases. We conclude that *happy* is easier to recognize for F-based models in all countries when no facial deformations caused by speaking occur. However, for SP, we observe an increase in confusion between *neutral* and *pensive* when training with WH, which might be explained by a difference in user behavior or annotation procedure between SP and the other countries when users do not speak. This, in turn, might explain their difference in *neutral-pensive* ratios (61:1 for SP, 33:1 for FR, and 17:1 for NO), compared to their similarity when users speak (around 5:1 for all countries). By contrast, NO shows a different trend compared to the other countries for *pensive* with G when training with WH: for SP and FR, *pensive* accuracy decreases when training with silence and it is maintained when training with speech and silence, while for NO, its accuracy increases for both cases. However, we observe again that this accuracy increase also comes with a slight increase in confusion with *happy*. The performance decrease for *pensive* for FR and SP is opposed to the general increase observed for this class on video-under-speech.

6) *Expanding training data including speech instances:* As can be seen in Table VII, this training regime slightly reduces the performance of all models on average when applied on WH except for G, for which accuracy is maintained but stability decreases. For F-based models, we observe an increase in confusion between *neutral* and the other two classes since the facial deformations added during training decrease discriminative power, while for G, confusion with *neutral* mostly increases for *happy*. Fig. 5b allows us to compare cross-country performance. We observe similar trends as for WH on average and class-wise, except for SP→SP with *happy* and NO→NO. The difference in overall behavior for NO is unclear. We observe a general performance decrease for *pensive*, which could be caused by the high difference in the number of instances between speech and silence sets (808% increase for WH when including speech instances, and similarly high per country), causing the models to learn patterns more tailored to *pensive* episodes while speaking. This issue may also be causing part of the performance deterioration for *happy*, and it is the opposite effect observed for the video-under-speech scenario when training on all data. In general, and in contrast to video-under-speech, per-class accuracies

tend to be more balanced when trained on silence only for F-
based models, but also when trained on WH. For G, it is harder
to categorize due to *pensive* being the main discriminated
class. Nonetheless, this auxiliary modality tends to perform
best on average when training on all data, indicating that it
benefits from the added variability from different countries
and speaking status, regardless of the evaluation scenario.

7) *Effect of speaking status on video-based evaluation:* For WH, the users’ emotional expressions can be better recognized when they are not speaking with G and F-based models on average. Indeed, evaluating on silence instances with no facial deformations that add noise should increase the discriminative power for *happy* and *pensive*. However, G should not be directly affected by facial deformations, so this leads us to believe that gaze and head patterns may be correlated to speaking status beyond facial deformations, and that such patterns are more discriminative for silence instances. Country-wise, however, trends differ for F-based models depending on the country and the training regime. Class-wise, *neutral* always obtains the highest accuracy for silence samples trained on silence and speech per country, along with a higher number of false positives. *Happy* is easier to recognize during silence instances for SP and FR with F when training on WH with silent instances. For NO, *happy* is easier to recognize during speech instances with models that combine F and A, also when training with WH, highlighting again the importance of A for this country. For WH, both scenarios are equally performant. Finally, *pensive* is easier to discriminate when evaluating on silence instances for FR, SP, and WH, although it is difficult to determine which training regime is best due to fluctuating confusion patterns. By contrast, for NO, this class is easier to categorize when the user is speaking with F+G when training on WH with speech instances. Note that, with respect to sample size, both speech and silence subsets are given in around 1:1 ratio for *neutral* and 2:1 for *happy*, while for *pensive*, ratios range from 14:1 for SP to 4:1 for NO.

VI. DISCUSSION

Here, we summarize the findings for each research question
(Sec. V-A), and discuss limitations and potential future work.

1) *Discriminative power of the main modality:* We observe varying levels of confusion between the minority classes and the majority class for both label types, with the highest confusion being observed with *pensive*. We strongly believe that this is mostly caused by the FER model considering only spatial information, since this class is mostly characterized by specific dynamics. Consequently, it is the most benefited when F is combined with G, which does provide salient dynamics.

Nevertheless, the minority classes tend to be well dis-
criminated against each other. Despite employing a balancing
strategy, per-class accuracies seem to be associated with their
corresponding sample sizes. It is important to note, however,
that the majority class in both label types (*neutral* and *calm*)
is characterized by a relatively lower intensity or absence of
emotional expression and can be encountered in transitional
phases between other emotions. This poses a challenge in
establishing clear category boundaries. All things considered,
average accuracies are around 60-70%, which is a decent
rate for a three-class unimodal classifier given the nature of
our scenario. This could be further improved by leveraging
temporal dependencies across time steps [101].

2) *Performance of auxiliary modalities compared to the main modality:* This depends on the label type. For the audio-based scenario, G struggles to achieve sufficient accuracy. By contrast, F does a better job, usually recognizing *pleased* with similar or higher accuracy and discrimination power than A alone, particularly due to the similarity between *happy* and *pleased* facial features. Conversely, for the video-based scenario, both A and G achieve accuracies closer to the main modality, especially when evaluating on silence instances. In particular, G alone is able to recognize *pensive* with extremely high accuracy, always better than F alone. In addition, in some cases, A is capable of obtaining higher accuracies than F for *pensive* and *happy*, being especially informative for the latter.

Achieving such recognition rates using only the auxiliary modalities is advantageous in scenarios where network or sensor failures may produce potential asynchronies between audio and video streams or even data loss. Having channel-specific labels already allows for individual processing. Therefore, if one stream is affected or even deactivated to reduce processing times, the system can still maintain functionality by utilizing the auxiliary modalities. In addition, the video modality may be intentionally altered or disabled for privacy purposes. In the case of alteration, G could still be extracted; however, in the event of video deactivation, only A would remain useful, from which the important video-based events could still be recognized. Nonetheless, as a prospective direction, it is worth considering crossmodal training techniques [55], which learn from multiple modalities at training time to improve single-modality recognition during inference.

3) *Multimodality:* It is beneficial, provided that the modality that is added to the main modality provides discriminative information for a given class. This is the case when adding F to A for the audio-based scenario for the minority classes, and when adding A or G to F for the video-based scenario for all classes, except *happy* for G. The result also depends on the distribution of features: the number of network parameters usually increases with the number of input features, which may lead to overfitting if not accounted for.

The increase in accuracy when combining A and F is in line
with the large audiovisual emotion recognition literature [4]
and with the few works addressing our target population [18].
It is difficult to contextualize the A/F+G results within the
literature, as the conclusions depend on the scenario and
features used, and most speech-video-gaze/head works employ
the VAD model instead [53]. As an example of discrete
emotion recognition, our results resemble the findings of the
aforementioned crossmodal work [55], for which including
gaze features also improved performance for video-test, while
for audio-test only one of their gaze alternatives outperforms
the no-gaze option. Class-wise, they observed a similar gaze
spatial distribution between *neutral* and *happy*.

Ideally, however, a multimodal system should effectively
disregard irrelevant information from individual modalities,
thereby preventing any detrimental impact on performance.

In that sense, simple feature concatenation may not be a
solution to this problem, as it may fail to capture the interac-
tions between features adequately. By contrast, attention-based
methods are known to adaptively balance the contributions of
different modalities [68]. Nonetheless, our preliminary exper-
iments found no differences among fusion types. Although it
is possible that both approaches do yield similar results in our
scenario, 1) the stage-wise training, and 2) the fixed-modality
synchronization applied are factors that may have influenced
the outcome. For the former, end-to-end training would allow
modality features to jointly evolve from early network lay-
ers [101]. For the latter, a flexible temporal synchronization
able to capture long-term cross-modal dependencies would
compensate for the unaligned nature of communication [102].

4) *Cross-country differences:* There are numerous differ-
ences that have been previously discussed and will not be
reiterated here, exposing disparities in how emotions are
expressed and perceived across countries. Such behavior and
annotation differences are also reflected in the divergence of
the proportions of per-class sample sizes and distribution in the
feature space. Nonetheless, results are subject to the country-
wise sample imbalance. This causes NO, the country with the
lowest sample size, to obtain lower performance in general.

5) *Multi-country training:* In general, main modalities and multimodal combinations benefit from this training regime. With respect to auxiliary modalities, however, only G benefits from it for the video-based scenario, indicating that the additional features are not as transferable cross-country as the main modalities. Performance gains mainly come from the increased variability, which especially benefits minority classes. However, the extent of such gains does not correlate with the sample size increase, since such increase is proportionally similar for the audio-based scenario than for the video-based scenario, but gains are higher for the latter. This can be attributed to: 1) the audio-based scenario having fewer samples than video; 2) A features already being more generic since they come from a large-scale model, thus more variability may not affect results; 3) some acoustic features of emotional expressions being more language- or culture-specific and hence less transferable across countries than facial expressions, which is in line with previous studies [82]. Note that no sampling strategy has been applied to balance the number of data samples across countries, hence having a bias towards SP, the country with the highest number of samples. Nonetheless, as found in our experiments, the effect of multi-country training will always depend on the feature distribution and the cultural similarities across countries.

6) *Training with spoken and silent instances:* Training on all data consistently but marginally improves performance when evaluating on spoken instances, increasing discriminative power for the minority classes. This is due to the fact that the added variability, without the noise produced by speaking, helps the models learn discriminative features that are less influenced by the speaking effect. Interestingly though, the performance obtained with this training regime is on par with that obtained when combining F and A trained on spoken instances only. By contrast, when evaluating on silent instances, training with all data produces the opposite effect, as
the variability added makes the recognition task harder. Note
that there is no sampling strategy to balance the number of
spoken/silent data, hence having a slight bias towards spoken
samples, primarily for the minority classes. Nonetheless, these
findings indicate that speaking status significantly influences
the learning process and should be taken into account when
devising similar solutions to ours.

7) *Speech vs silence performance:* This is highly country-specific. In general, and related to RQ₆, both speaking statuses achieve similar accuracy on average, but the minority classes tend to be better discriminated against when the user is not speaking, provided the model was trained on silent instances only (note that the original F features were trained on both instance types, so differences in performance will come exclusively from the final models). This conclusion is intuitive for F-based models since facial deformations affect discriminative power. However, since this difference in performance is also observed for G, this indicates that gaze and head patterns are correlated with speaking status beyond facial deformations, and such patterns are more discriminative for silence instances. Nonetheless, if A is combined with F for spoken instances, both spoken and silent instances can obtain comparable results.

8) *Audio vs video performance:* Audio-based results are constrained by the limited amount of data. While this effect is balanced for A by the use of a pretrained large-scale model, which may also aid in generalization ability, auxiliary modalities are indeed impacted. By contrast, the video-based evaluation can leverage sample sizes two orders of magnitude higher, but with higher redundancy among samples. Overall, we observe a higher performance for video-based evaluation than for audio. This can be attributed to the differences in the number of samples. Another reason could be that WavLM was trained on English speech; therefore, even if the extracted features are generic enough due to the large pretraining, they are not specialized to the languages considered in this work and hence important emotional nuances might have been lost.

VII. CONCLUSION

We presented a first comprehensive study on non-verbal
emotion expression recognition in interactions between older
adults and a simulated VC within the context of the EM-
PATHIC project. We described the rationale for data collection
and annotation procedure aimed at developing a computational
approach that could leverage cues from audio and video
channels separately. By analyzing the influence of different
modalities, training approaches, and communication modes,
we aimed to shed light on some of the factors that affect the
effectiveness of emotion recognition in this scenario. Our find-
ings show that facial, speech, head, and gaze cues contribute
to the accurate recognition of the channel-specific expressions
considered with varying levels of discriminative power. As the
evaluation was conducted in a subject-independent manner,
these cues would prove even more valuable for a personalized
online setup, in which the model could continuously learn
from the user's behavior during the interaction. In addition,
we determined that multi-country training can generally com-
pensate for limited country-specific data, thereby enhancing
overall performance despite cross-country differences.

The insights gained are expected to contribute to the de-
velopment of more accurate emotion recognition systems, and
pave the way for improved VC experiences and personalized
technologies catering to the well-being of this age group.

ACKNOWLEDGMENTS

This work has been partially supported by the European
Union's Horizon 2020 R&I program under grant agreement No
769872 (EMPATHIC project), the Spanish project PID2022-
136436NB-I00, and ICREA under ICREA Academia program.

REFERENCES

[1] A. Jaimes and N. Sebe, “Multimodal human–computer interaction: A survey,” *Computer vision and image understanding*, vol. 108, no. 1-2, pp. 116–134, 2007.

[2] R. Justo, L. Ben Letaifa, C. Palmero, E. Gonzalez-Fraile, A. Torp Johansen, A. Vázquez, G. Cordasco, S. Schlögl, B. Fernández-Ruanova, M. Silva, S. Escalera, M. deVelasco, J. Tenorio-Laranga, A. Esposito, M. Korsnes, and M. I. Torres, “Analysis of the interaction between elderly people and a simulated virtual coach,” *Journal of Ambient Intelligence and Humanized Computing*, vol. 11, pp. 6125–6140, 2020.

A. Vázquez, A. López Zorrilla, J. M. Olaso, and M. I. Torres, “Dialogue management and language generation for a robust conversational virtual coach: Validation and user study,” *Sensors*, vol. 23, no. 3, 2023.

[4] S. Poria, E. Cambria, R. Bajpai, and A. Hussain, "A review of affective computing: From unimodal analysis to multimodal fusion," *Information fusion*, vol. 37, pp. 98–125, 2017.

[5] P. V. Rouast, M. T. Adam, and R. Chiong, “Deep learning for human affect recognition: Insights and new developments,” *IEEE Transactions on Affective Computing*, vol. 12, no. 2, pp. 524–543, 2019.

[6] S. K. D'mello and J. Kory, "A review and meta-analysis of multimodal affect detection systems," *ACM computing surveys*, vol. 47, no. 3, pp. 1–36, 2015.

[7] B. Schuller, F. Weninger, Y. Zhang, F. Ringeval, A. Batliner, S. Steidl, F. Eyben, E. Marchi, A. Vinciarelli, K. Scherer, M. Chetouani, and M. Mortillaro, "Affective and behavioural computing: Lessons learnt from the first computational paralinguistics challenge," *Computer Speech & Language*, vol. 53, pp. 156–180, 2019.

[8] R. Chakraborty, M. Pandharipande, and S. K. Kopparapu, *Analyzing emotion in spontaneous speech*. Springer, 2017.

[9] M. deVelasco, R. Justo, and M. I. Torres, “Automatic identification of emotional information in spanish tv debates and human-machine interactions,” *Applied Sciences*, vol. 12, no. 4, 2022.

[10] S. Mariooryad and C. Busso, "Facial expression recognition in the presence of speech using blind lexical compensation," *IEEE Transactions on Affective Computing*, vol. 7, no. 4, pp. 346–359, 2015.

[11] Z. Zeng, M. Pantic, G. Roisman, and T. Huang, “A survey of affect recognition methods: Audio, visual, and spontaneous expressions,” *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 1, no. 31, pp. 39–58, 2009.

[12] J. M. Olaso, M. García-Sebastián, A. López Zorrilla, M. Tainta, M. Ecay-Torres, M. Torres, and P. Martínez-Lage, “The cita go-on dialogue system: Mid-term achievements,” in *Proc. ACM Int. Conf. on PErvasive Technologies Related to Assistive Environments*, 2023.

[13] M. McTear, K. Jokinen, M. M. Alam, Q. Saleem, G. Napolitano, F. Szczepaniak, M. Hariz, G. Chollet, C. Lohr, J. Boudy, Z. Azimi, S. D. Roelen, and R. Wieching, “Interaction with a virtual coach for active and healthy ageing,” *Sensors*, vol. 23, no. 5, 2023.

[14] A. Demaeght, C. Miclau, J. Hartmann, J. Markwardt, and O. Korn, "Multimodal emotion analysis of robotic assistance in elderly care," in *Proc. of the 15th Int. Conf. on PErvasive Technologies Related to Assistive Environments*, 2022, pp. 230–236.

[15] C. Magai, N. S. Consedine, Y. S. Krivoshekova, E. Kudadjie-Gyamfi, and R. McPherson, "Emotion experience and expression across the adult life span: insights from a multimodal assessment study." *Psychology and aging*, vol. 21, no. 2, p. 303, 2006.

[16] M. Fölster, U. Hess, and K. Werheid, “Facial age affects emotional expression decoding,” *Frontiers in Psychology*, vol. 5, 2014.

[17] R. W. Levenson, L. L. Carstensen, W. V. Friesen, and P. Ekman, "Emotion, physiology, and expression in old age." *Psychology and aging*, vol. 6, no. 1, p. 28, 1991.
[18] K. Ma, X. Wang, X. Yang, M. Zhang, J. M. Girard, and L.-P. Morency, "Elderreact: a multimodal dataset for recognizing emotional response in aging adults," in *Int. Conf. on Multimodal Interaction*, 2019, pp. 349–357.

[19] K. Wang, N. An, B. N. Li, Y. Zhang, and L. Li, "Speech emotion recognition using fourier parameters," *IEEE Transactions on affective computing*, vol. 6, no. 1, pp. 69–75, 2015.

[20] N. Lopes, A. Silva, S. R. Khanal, A. Reis, J. Barroso, V. Filipe, and J. Sampaio, "Facial emotion recognition in the elderly using a svm classifier," in *2nd Int. Conf. on Technology and Innovation in Sports, Health and Wellbeing*. IEEE, 2018, pp. 1–5.

[21] B. W. Schuller, A. Batliner, C. Bergler, E.-M. Messner, A. Hamilton, S. Amiriparian, A. Baird, G. Rizos, M. Schmitt, L. Stappen, H. Baumeister, A., Deighton MacIntyre, and S. Hantke, "The interspeech 2020 computational paralinguistics challenge: Elderly emotion, breathing & masks," 2020.

[22] J. Kossaifi, R. Walecki, Y. Panagakis, J. Shen, M. Schmitt, F. Ringeval, J. Han, V. Pandit, A. Toisoul, B. Schuller, H. E. Star, Kam, and M. Pantic, "Sewa db: A rich database for audio-visual emotion and sentiment research in the wild," *IEEE transactions on pattern analysis and machine intelligence*, vol. 43, no. 3, pp. 1022–1040, 2019.

[23] M. I. Torres, J. M. Olaso, C. Montenegro, R. Santana, A. Vázquez, R. Justo, J. A. Lozano, S. Schlögl, G. Chollet, N. Dugan, M. Irvine, N. Glackin, C. Pickard, A. Esposito, G. Cordasco, A. Troncone, D. Petrovska-Delacretaz, A. Mtibaa, M. A. Hmani, M. S. Korsnes, L. J. Martinussen, S. Escalera, C. P. Cantariño, O. Deroo, O. Gordeeva, J. Tenorio-Laranga, E. Gonzalez-Fraile, B. Fernandez-Ruanova, and A. Gonzalez-Pinto, "The empathic project: Mid-term achievements," in *Proc. of the 12th ACM Int. Conf. on Pervasive Technologies Related to Assistive Environments*, 2019, p. 629–638.

[24] J. M. Olaso, A. Vázquez, L. Ben Letaifa, M. de Velasco, A. Mtibaa, M. A. Hmani, D. Petrovska-Delacretaz, G. Chollet, C. Montenegro, A. López-Zorrilla, R. Justo, R. Santana, J. Tenorio-Laranga, E. González-Fraile, B. n. Fernández-Ruanova, G. Cordasco, A. Esposito, K. B. Gjellesvik, A. T. Johansen, M. S. Kornes, C. Pickard, C. Glackin, G. Cahalane, P. Buch, C. Palmero, S. Escalera, O. Gordeeva, O. Deroo, A. Fernández, D. Kylslitska, J. A. Lozano, M. I. Torres, and S. Schlögl, "The empathic virtual coach: A demo," in *Int. Conf. on Multimodal Interaction*, 2021, p. 848–851.

[25] P. Ekman, "Basic emotions," *Handbook of cognition and emotion*, vol. 98, no. 45-60, p. 16, 1999.

[26] H. Gunes and M. Pantic, "Automatic, dimensional and continuous emotion recognition," *International Journal of Synthetic Emotions*, vol. 1, no. 1, pp. 68–99, 2010.

[27] R. A. Calvo and S. D'Mello, "Affect detection: An interdisciplinary review of models, methods, and their applications," *IEEE Transactions on affective computing*, vol. 1, no. 1, pp. 18–37, 2010.

[28] B. Schuller, A. Batliner, S. Steidl, and D. Seppi, "Recognising realistic emotions and affect in speech: State of the art and lessons learnt from the first challenge," *Speech communication*, vol. 53, no. 9-10, pp. 1062–1087, 2011.

[29] J. A. Russell, "A circumplex model of affect," *Journal of personality and social psychology*, vol. 39, no. 6, p. 1161, 1980.

[30] M. Valstar, B. Schuller, K. Smith, T. Almaev, F. Eyben, J. Krajewski, R. Cowie, and M. Pantic, "Avec 2014: 3d dimensional affect and depression recognition challenge," in *Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge*. ACM, 2014, pp. 3–10.

[31] K.-Y. Huang, C.-H. Wu, Q.-B. Hong, M.-H. Su, and Y.-H. Chen, "Speech emotion recognition using deep neural network considering verbal and nonverbal speech sounds," in *IEEE Int. Conf. on Acoustics, Speech and Signal Processing*, 2019, pp. 5866–5870.

[32] M. de Velasco, "Analysis and Automatic Identification of Spontaneous Emotions in Speech from Human-Human and Human-Machine Communication," Ph.D. dissertation, Departamento de Electricidad y Electrónica. Universidad del País Vasco UPV/EHU, 2023.

[33] B. Schuller, S. Steidl, A. Batliner, A. Vinciarelli, K. Scherer, F. Ringeval, M. Chetouani, F. Weninger, F. Eyben, E. Marchi, M. Mortillaro, H. Salamin, A. Polychroniou, F. Valente, and S. Kim, "The interspeech 2013 computational paralinguistics challenge: Social signals, conflict, emotion, autism," in *Proc. 14th Annual Conf. of the International Speech Communication Association*, 2013.

[34] F. Eyben, K. R. Scherer, B. W. Schuller, J. Sundberg, E. André, C. Busso, L. Y. Devillers, J. Epps, P. Laukka, S. S. Narayanan, and K. P. Truong, "The geneva minimalistic acoustic parameter set (gemaps) for voice research and affective computing," *IEEE transactions on affective computing*, vol. 7, no. 2, pp. 190–202, 2015.

[35] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, "wav2vec 2.0: A framework for self-supervised learning of speech representations," *Advances in neural information processing systems*, vol. 33, pp. 12449–12460, 2020.

[36] C. Luna-Jiménez, R. Kleinlein, D. Griol, Z. Callejas, J. Montero, and F. Fernández-Martínez, "A proposal for multimodal emotion recognition using aural transformers and action units on ravdess dataset," *Applied Sciences*, vol. 12, 2022.

[37] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, "Hubert: Self-supervised speech representation learning by masked prediction of hidden units," *IEEE/ACM Trans. on Audio, Speech, and Language Processing*, vol. 29, pp. 3451–3460, 2021.

[38] C. Wang, Y. Wu, Y. Qian, K. Kumatani, S. Liu, F. Wei, M. Zeng, and X. Huang, "Unispeech: Unified speech representation learning with labeled and unlabeled data," in *Int. Conf. on Machine Learning*. PMLR, 2021, pp. 10937–10947.

[39] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y. Qian, Y. Qian, J. Wu, M. Zeng, X. Yu, and F. Wei, "Wavlm: Large-scale self-supervised pre-training for full stack speech processing," *IEEE Journal of Selected Topics in Signal Processing*, vol. 16, no. 6, pp. 1505–1518, 07 2022.

[40] Y. B. Singh and S. Goel, "A systematic literature review of speech emotion recognition approaches," *Neurocomputing*, vol. 492, pp. 245–263, 2022.

[41] J. de Lope and M. Graña, "An ongoing review of speech emotion recognition," *Neurocomputing*, vol. 528, pp. 1–11, 2023.

[42] J. Wagner, A. Triantafyllopoulos, H. Wierstorf, M. Schmitt, F. Burkhardt, F. Eyben, and B. W. Schuller, "Dawn of the transformer era in speech emotion recognition: closing the valence gap," *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 2023.

[43] B. T. Atmaja, A. Sasou, and M. Akagi, "Survey on bimodal speech emotion recognition from acoustic and linguistic information fusion," *Speech Communication*, vol. 140, pp. 11–28, 2022.

[44] S. Li and W. Deng, "Deep facial expression recognition: A survey," *IEEE transactions on affective computing*, vol. 13, no. 3, pp. 1195–1215, 2020.

[45] C. Liu and H. Wechsler, "Gabor feature based classification using the enhanced fisher linear discriminant model for face recognition," *IEEE Transactions on Image Processing*, vol. 11, no. 4, pp. 467–476, 2002.

[46] C. Shan, S. Gong, and P. W. McOwan, "Facial expression recognition based on local binary patterns: A comprehensive study," *Image and Vision Computing*, vol. 27, no. 6, pp. 803–816, May 2009.

[47] S. M. Mavadati, M. H. Mahoor, K. Bartlett, P. Trinh, and J. F. Cohn, "DISFA: A spontaneous facial action intensity database," *IEEE Transactions on Affective Computing*, vol. 4, no. 2, pp. 151–160, 2013.

[48] C. A. Corneanu, M. O. Simón, J. F. Cohn, and S. E. Guerrero, "Survey on rgb, 3d, thermal, and multimodal approaches for facial expression recognition: History, trends, and affect-related applications," *IEEE transactions on pattern analysis and machine intelligence*, vol. 38, no. 8, pp. 1548–1568, 2016.

[49] B. C. Ko, "A brief review of facial emotion recognition based on visual information," *sensors*, vol. 18, no. 2, p. 401, 2018.

[50] A. Mollahosseini, D. Chan, and M. H. Mahoor, "Going deeper in facial expression recognition using deep neural networks," in *IEEE Winter conference on applications of computer vision*, 2016, pp. 1–10.

[51] R. Graham and K. S. LaBar, "Neurocognitive mechanisms of gaze-expression interactions in face processing and social attention," *Neuropsychologia*, vol. 50, no. 5, pp. 553–566, 2012.

[52] J. Z. Lim, J. Mountstephens, and J. Teo, "Emotion recognition using eye-tracking: taxonomy, review and current challenges," *Sensors*, vol. 20, no. 8, p. 2384, 2020.

[53] J. O'Dwyer, N. Murray, and R. Flynn, "Affective computing using speech and eye gaze: a review and bimodal system proposal for continuous affect prediction," *arXiv preprint arXiv:1805.06652*, 2018.

[54] S. Ghosh, A. Dhall, M. Hayat, J. Knibbe, and Q. Ji, "Automatic gaze analysis: A survey of deep learning based approaches," *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 46, no. 1, pp. 61–84, 2023.

[55] A. Abdou, E. Sood, P. Müller, and A. Bulling, "Gaze-enhanced cross-modal embeddings for emotion recognition," *Proc. ACM on Human-Computer Interaction*, vol. 6, no. ETRA, pp. 1–18, 2022.

[56] K. Cortacero, T. Fischer, and Y. Demiris, "Rt-bene: A dataset and baselines for real-time blink estimation in natural environments," in *Proc. International Conference on Computer Vision Workshops*, 2019.

[57] R. El Kaliouby and P. Robinson, "Real-time inference of complex mental states from facial expressions and head gestures," *Real-time vision for human-computer interaction*, pp. 181–200, 2005.
[58] U. Hess, R. B. Adams, and R. E. Kleck, "Looking at you or looking elsewhere: The influence of head orientation on the signal value of emotional facial expressions," *Motivation and Emotion*, vol. 31, pp. 137-144, 2007.

[59] C. Busso, Z. Deng, M. Grimm, U. Neumann, and S. Narayanan, "Rigid head motion in expressive speech animation: Analysis and synthesis," *IEEE Transactions on audio, speech, and language processing*, vol. 15, no. 3, pp. 1075-1086, 2007.

[60] H. Gunes and M. Pantic, "Dimensional emotion prediction from spontaneous head gestures for interaction with sensitive artificial listeners," in *Intelligent Virtual Agents: 10th Int. Conf., IVA. Proceedings 10*. Springer, 2010, pp. 371-377.

[61] A. Adams, M. Mahmoud, T. Baltrušaitis, and P. Robinson, "Decoupling facial expressions and head motions in complex emotions," in *Int. Conf. on affective computing and intelligent interaction*, 2015, pp. 274-280.

[62] Y. Ding, L. Shi, and Z. Deng, "Low-level characterization of expressive head motion through frequency domain analysis," *IEEE Transactions on Affective Computing*, vol. 11, no. 3, pp. 405-418, 2018.

[63] A. Samanta and T. Guha, "Emotion sensing from head motion capture," *IEEE Sensors Journal*, vol. 21, no. 4, pp. 5035-5043, 2020.

[64] K. Khan, R. U. Khan, R. Leonardi, P. Migliorati, and S. Benini, "Head pose estimation: A survey of the last ten years," *Signal Processing: Image Communication*, vol. 99, p. 116479, 2021.

[65] T. Xue, A. E. Ali, G. Ding, and P. Cesar, "Investigating the relationship between momentary emotion self-reports and head and eye movements in hmd-based 360 vr video watching," in *Extended abstracts of the 2021 conference on human factors in computing systems*, 2021, pp. 1-8.

[66] J. O'Dwyer, N. Murray, and R. Flynn, "Eye-based continuous affect prediction," in *8th Int. Conf. on Affective Computing and Intelligent Interaction*. IEEE, 2019, pp. 137-143.

[67] T. Baltrušaitis, C. Ahuja, and L.-P. Morency, "Multimodal machine learning: A survey and taxonomy," *IEEE transactions on pattern analysis and machine intelligence*, vol. 41, no. 2, pp. 423-443, 2018.

[68] W. Guo, J. Wang, and S. Wang, "Deep multimodal representation learning: A survey," *IEEE Access*, vol. 7, pp. 63 373-63 394, 2019.

[69] C.-H. Wu, J.-C. Lin, and W.-L. Wei, "Survey on audiovisual emotion recognition: databases, features, and data fusion strategies," *APSIPA transactions on signal and information processing*, vol. 3, p. e12, 2014.

[70] J. F. Cohn, L. I. Reed, T. Moriyama, J. Xiao, K. Schmidt, and Z. Ambadar, "Multimodal coordination of facial action, head rotation, and eye motion during spontaneous smiles," in *IEEE Int. Conf. on Automatic Face and Gesture Recognition*, 2004, pp. 129-135.

[71] S. Wu, Z. Du, W. Li, D. Huang, and Y. Wang, "Continuous emotion recognition in videos by fusing facial expression, head pose and eye gaze," in *Int. Conf. on Multimodal Interaction*, 2019, pp. 40-48.

[72] A. Alhargan, N. Cooke, and T. Binjammaz, "Multimodal affect recognition in an interactive gaming environment using eye tracking and speech signals," in *Proc. ACM Int. Conf. on multimodal interaction*, 2017, pp. 479-486.

[73] J. O'Dwyer, "Speech, head, and eye-based cues for continuous affect prediction," in *8th Int. Conf. on Affective Computing and Intelligent Interaction Workshops and Demos*. IEEE, 2019, pp. 16-20.

[74] H. Abdollahi, M. H. Mahoor, R. Zandie, J. Siewierski, and S. H. Qualls, "Artificial emotional intelligence in socially assistive robots for older adults: a pilot study," *IEEE Transactions on Affective Computing*, vol. 14, no. 3, pp. 2020-2032, 2022.

[75] P. Sreevidya, S. Veni, and O. Ramana Murthy, "Elder emotion classification through multimodal fusion of intermediate layers and cross-modal transfer learning," *Signal, Image and Video Processing*, vol. 16, no. 5, pp. 1281-1288, 2022.

[76] S. R. Jannat and S. Canavan, "Expression recognition across age," in *Int. Conf. on Automatic Face and Gesture Recognition*, 2021, pp. 1-5.

[77] S. Schlögl, G. Doherty, and S. Luz, "Wizard of oz experimentation for language technology applications: challenges and tools." *Interacting with Computers*, vol. 27, no. 6, pp. 592-615, Nov 2015.

[78] A. S. Cowen and D. Keltner, "Self-report captures 27 distinct categories of emotion bridged by continuous gradients," *Proc. of the national academy of sciences*, vol. 114, no. 38, pp. E7900-E7909, 2017.

[79] A. Esposito, "The amount of information on emotional states conveyed by the verbal and nonverbal channels: Some perceptual data," in *Progress in nonlinear speech processing*. Springer, 2007, pp. 249-268.

[80] —, "The perceptual and cognitive role of visual and auditory channels in conveying emotional information," *Cognitive Computation*, vol. 1, pp. 268-278, 2009.

[81] J. A. Russell, J.-A. Bachorowski, and J.-M. Fernández-Dols, "Facial and vocal expressions of emotion," *Annual review of psychology*, vol. 54, no. 1, pp. 329-349, 2003.

[82] M. T. Riviello and A. Esposito, "A cross-cultural study on the effectiveness of visual and vocal channels in transmitting dynamic emotional information," *Acta Polytech. Hung.*, vol. 9-1, no. 1, pp. 157-170, 2012.

[83] Q. Du, S. Labat, T. Demeester, and V. Hoste, "Unimodalities count as perspectives in multimodal emotion annotation," in *Proc. 2nd Workshop on Perspectivist Approaches to NLP co-located with 26th European Conference on Artificial Intelligence*, 2023.

[84] J. Sun, S. Han, Y.-P. Ruan, X. Zhang, S.-K. Zheng, Y. Liu, Y. Huang, and T. Li, "Layer-wise fusion with modality independence modeling for multi-modal emotion recognition," in *Proc. Annual Meeting of the Assoc. for Computational Linguistics*, 2023, pp. 658-670.

[85] M. L. McHugh, "Interrater reliability: the kappa statistic," *Biochemia medica*, vol. 22, no. 3, pp. 276-282, 2012.

[86] S. Steininger, F. Schiel, O. Dioubina, and S. Raubold, "Development of user-state conventions for the multimodal corpus in smartkom," in *Proc. Workshop on Multimodal Resources and Multimodal Systems Evaluation*. Citeseer, 2002, pp. 33-37.

[87] C. Greco, C. Buono, P. Buch-Cardona, G. Cordasco, S. Escalera, A. Esposito, A. Fernandez, D. Kyslitska, M. S. Kornes, C. Palmero, J. Tenorio-Laranga, A. Torp Johansen, and M. I. Torres, "Emotional features of interactions with empathic agents," in *Proc. IEEE/CVF International Conference on Computer Vision*, 2021, pp. 2168-2176.

[88] J. Li, "Emotion and ageing in discourse: Do older people express more positive emotions?" *Corpus-based Studies across Humanities*, vol. 1, no. 1, pp. 23-50, 2024.

[89] S. Zhang, X. Zhu, Z. Lei, H. Shi, X. Wang, and S. Z. Li, "Faceboxes: A cpu real-time face detector with high accuracy," in *IJCB*, 2017.

[90] J. Guo, X. Zhu, Y. Yang, F. Yang, Z. Lei, and S. Z. Li, "Towards fast, accurate and stable 3d dense face alignment," in *European Conference on Computer Vision*. Springer, 2020, pp. 152-168.

[91] F. Chollet, "Xception: Deep learning with depthwise separable convolutions," in *Proc. IEEE conference on computer vision and pattern recognition*, 2017, pp. 1251-1258.

[92] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "ImageNet: A large-scale hierarchical image database," in *IEEE Conference on Computer Vision and Pattern Recognition*. IEEE, Jun. 2009.

[93] P. Huber, G. Hu, R. Tena, P. Mortazavian, P. Koppen, W. J. Christmas, M. Ratsch, and J. Kittler, "A multiresolution 3d morphable face model and fitting framework," in *Proceedings of the 11th international joint conference on computer vision, imaging and computer graphics theory and applications*, 2016.

[94] V. Lepetit, F. Moreno-Noguer, and P. Fua, "Epnp: An accurate o(n) solution to the pnp problem," *International Journal Of Computer Vision*, vol. 81, pp. 155-166, 2009.

[95] X. Zhang, S. Park, T. Beeler, D. Bradley, S. Tang, and O. Hilliges, "Eth-xgaze: A large scale dataset for gaze estimation under extreme head pose and gaze variation," in *European Conference on Computer Vision*. Springer, 2020, pp. 365-381.

[96] A. Bulling, J. A. Ward, H. Gellersen, and G. Tröster, "Eye movement analysis for activity recognition using electrooculography," *IEEE transactions on pattern analysis and machine intelligence*, vol. 33, no. 4, pp. 741-753, 2010.

[97] T. Amorese, C. Greco, M. Cuciniello, C. Buono, C. Palmero, P. Buch-Cardona, S. Escalera, M. I. Torres, G. Cordasco, and A. Esposito, "Using eye tracking to investigate interaction between humans and virtual agents," in *IEEE Conference on Cognitive and Computational Aspects of Situation Management*, 2022, pp. 125-132.

[98] V. Rajan, A. Brutti, and A. Cavallaro, "Is cross-attention preferable to self-attention for multi-modal emotion recognition?" in *IEEE Int. Conf. on Acoustics, Speech and Signal Processing*, 2022, pp. 4693-4697.

[99] R. R. Bouckaert and E. Frank, "Evaluating the replicability of significance tests for comparing learning algorithms," in *Pacific-Asia Conference on Knowledge Discovery and Data Mining*, vol. 3056. Springer, 2004, pp. 3-12.

[100] Y. Benjamini, A. M. Krieger, and D. Yekutieli, "Adaptive linear step-up procedures that control the false discovery rate," *Biometrika*, vol. 93, no. 3, pp. 491-507, 2006.

[101] P. Tzirakis, G. Trigeorgis, M. A. Nicolaou, B. W. Schuller, and S. Zafeiriou, "End-to-end multimodal emotion recognition using deep neural networks," *IEEE Journal of selected topics in signal processing*, vol. 11, no. 8, pp. 1301-1309, 2017.

[102] Y.-H. H. Tsai, S. Bai, P. P. Liang, J. Z. Kolter, L.-P. Morency, and R. Salakhutdinov, "Multimodal transformer for unaligned multimodal language sequences," in *Proc. Association for Computational Linguistics.*, 2019, p. 6558.