# THERADIA WoZ: An Ecological Corpus for Appraisal-based Affect Research in Healthcare

Hippolyte Fournier, Sina Alisamir, Safaa Azzakhnini, Isabella Zsoldos, Eléonore Trân, Patrick Constant, Gérard Bailly, Frédéric Elisei, Béatrice Bouchot, Brice Varini, et al.

► To cite this version:

Hippolyte Fournier, Sina Alisamir, Safaa Azzakhnini, Isabella Zsoldos, Eléonore Trân, et al. THERADIA WoZ: An Ecological Corpus for Appraisal-based Affect Research in Healthcare. IEEE Transactions on Affective Computing, 2025, pp.1-12. 10.1109/TAFFC.2025.3557465. hal-05046330

HAL Id: hal-05046330

https://hal.science/hal-05046330v1

Submitted on 25 Apr 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# THERADIA WoZ: An Ecological Corpus for Appraisal-based Affect Research in Healthcare

Hippolyte Fournier, Sina Alisamir, Safaa Azzakhnini, Isabella Zsoldos, Eléonore Trân, Gérard Bailly, Frédéric Elisei, Béatrice Bouchot, Brice Varini, Patrick Constant, Joan Fruitet, Franck Tarpin-Bernard, Solange Rossato, François Portet, Olivier Koenig, Hanna Chainay, Fabien Ringeval

**Abstract**—We present THERADIA WoZ, an ecological corpus designed for audiovisual research on affect in healthcare. Two groups of senior individuals, consisting of 52 healthy participants and 9 individuals with Mild Cognitive Impairment (MCI), performed Computerised Cognitive Training (CCT) exercises while receiving support from a virtual assistant, tele-operated by a human in the role of a Wizard-of-Oz (WoZ). The audiovisual expressions produced by the participants were fully transcribed, and partially annotated based on dimensions derived from recent appraisal theory models, including novelty, intrinsic pleasantness, goal conduciveness, and coping. Additionally, the annotations included 23 affective labels from the literature of achievement affects. We present the data collection, transcription, and annotation protocols, alongside a detailed analysis of the annotated dimensions and labels. Baseline methods and results for their automatic prediction are also presented. Results reveal that the dimensions of appraisal theory can be predicted, with the performance varying across different modalities. The corpus aims to serve as a valuable resource for researchers in affective computing, and is made available to both industry and academia.

***Index Terms***—Ecological corpus, Computerised Cognitive Training, Appraisal theories, Dimensional / Categorical Affect Recognition

## I. INTRODUCTION

OVER the past decade, special attention has been given to the use of Artificial Intelligence (AI) technologies to improve healthcare. One of the rationales behind this craze is that the automation of tasks that do not necessarily require human intervention saves therapists time and mental load, resulting in more efficient practices and less burden for both patients and clinicians. However, recent reviews have evidenced that, even though AI technologies' performance can

<figure><img src="image_2.png" /><figcaption>Fig. 1. Head movements, gaze, speech, and articulation of a remote operator were captured in real-time to drive a virtual assistant as a Wizard-of-Oz [5].</figcaption></figure>

H. Fournier, S. Alisamir, S. Azzakhnini, F. Portet, S. Rossato, and F. Ringeval are with the Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LIG. E-mail: {hippolyte.fournier, sina.alisamir, safae.azzakhnini, francois.portet, solange.rossato, fabien.ringeval} @univ-grenoble-alpes.fr.

H. Chainay, O. Koenig, I. Zsoldos, and E. Trân are with the Univ. Lyon 2, EMC. E-mail: {hanna chainay, olivier.koenig, isabella.zsoldos, eleonore.tran} @univ-lyon2.fr.

G. Bailly and F. Elisei are with the GIPSA-lab, Univ. Grenoble Alpes. E-mail: {gerard.bailly, frederic.elisei} @gipsa-lab.grenoble-inp.fr.

B. Bouchot and B. Varini are with ATOS company. E-mail: {beatrice.bouchot, brice.varini} @atos.net.

P. Constant is with Pertimm company. E-mail: {patrick constant} @pertimm.com.

J. Fruitet and F. Tarpin-Bernard are with Humans Matter company. E-mail: {j.fruitet, f.tarpin} @humansmatter.co.

This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.

rival that of humans in certain specific tasks, such as serum analysis, or assessment of cardiovascular MRI images, they still struggle to deal with social interaction skills [1].

While the definition of social interaction skills remains a subject of debate [2], they can be summarised as the ability of autonomous agents to maintain equilibrium in a dynamic relationship with another agent, resulting in the development of autonomous relationships [3]. In the context of interactions between AI technology and patients in healthcare, it corresponds to the capabilities of systems to detect and respond appropriately to signals from patients that may disrupt the balance of interaction; e.g., patients may express that they feel misunderstood or unsatisfied. While verbally expressed requests are fairly well understood by today's AI technologies, it is still difficult for them to accurately detect patients' affective states from non-verbal signals [4].

Yet, the use of AI technologies for social interactions in healthcare would have far-reaching consequences. A prime illustration of their relevance lies in Computerised Cognitive Training (CCT), which aims to enhance or preserve cognitive functions in patients suffering from cognitive impairment through the repetition of exercises that target specific functions such as attention or memory [6]. Although the effectiveness of CCT has been evidenced by meta-analyses [6], [7], it seems to be conditioned by social interaction during the process [8], which requires the presence of the therapist. Thus, entrusting social interaction to AI technologies would be highly beneficial in making the process autonomous. To move in this direction, it is important to use meaningful data and reliable methods to accurately recognise patients' affective state in the context of CCT.
In this study, we present the THERADIA WoZ corpus, an ecological corpus specifically tailored for the audiovisual detection of affective states in the healthcare domain. The corpus was built to help the development of a virtual assistant supporting CCT sessions at home, for older healthy people and patients [5]. To the best of our knowledge, this is the first audiovisual corpus of affect in healthcare that fully relies on models of appraisal theories, while being accessible to both industrial and academic research communities<fnref n="1" />.

The corpus data come from natural interactions in French involving healthy and Mild Cognitive Impairment (MCI) senior participants. The interactions revolve around CCT exercises facilitated by a virtual assistant, and operated remotely by a human acting as a Wizard-of-Oz (WoZ), cf. Figure 1. Expressions of the participants were then fully transcribed and partially annotated based on the dimensions of recent appraisal theories models, along with labels derived from the literature of achievement affects.

A comprehensive analysis of the annotated labels was carried out, enabling the identification of a core set comprising ten affective states in the context of AI technology-human interactions in healthcare. This core set of affective labels was then used along with the appraisal dimensions to evaluate the performance of inference models based on audio, visual, and textual data.

The remainder of this paper is organised as follows. We first review the appraisal theory and annotation methods of affect, along with the existing corpora in healthcare in section II. The protocol used for the construction of our data set is then presented with the material and the participants in section III. The segmentation and the annotation of the recordings are explained in section IV. A comprehensive analysis of the affective annotations is then reported in section V, and results obtained in their automatic prediction are finally reported in section VI, before discussing the limitations of this study and concluding in section VII.

## II. STATE OF THE ART

In this section, we first review the definition of affective states and their annotation methods through the lens of appraisal theories, and then provide a concise overview of corpora of affect available in healthcare.

### A. Affective states in the lens of the appraisal theory

According to appraisal theories, affective phenomena are defined as multi-component processes aiming to maintain the well-being and survival of individuals in their environment [9]. Roughly speaking, affective phenomena arise when the meaning of environmental stimuli are cognitively appraised as potentially threatening to an individual's equilibrium in their environment.

This appraisal process would be divided into four criteria: the relevance of the events for the individual's well-being and/or survival, their implications, the possible coping with these implications, and their normative significance, i.e., the degree of matching with individual's norms and values [10].

Following this appraisal process, the activation of the organism's subsystems, i.e., physiological responses, action tendencies and motor expressions, would be synchronised to help individuals carry out the behaviours that contribute to the restoration of their homeostasis. In this perspective, an affective state serves as a marker reflecting how individuals appraise themselves between the plausible deregulation of their homeostasis caused by the environment, and the search for the behaviour leading to a return to equilibrium.

The cognitive appraisal process is supposed to be causal in the elicitation of affective episodes [11]. This position assumes a necessary and sufficient relationship would link appraisal to the synchronisation of the organism's subsystems [12]. In other words, a specific pattern of criteria of the appraisal process would trigger a specific synchronised activation of the organism's subsystems. Conversely, the observation of a specific synchronised activation would allow to predict the pattern of criteria of the appraisal process performed by the experimenter of the affective episode.

Based on these premises, the Tripartite Emotion Expression and Perception model postulates that the way humans perceive the affect of others is conditioned by this causal relationship [13]. The idea is that if an individual *A* is able to determine the affective state of an individual *B*, this is because *A* inferred – in part unconsciously – the cognitive appraisal process performed by *B* on the basis of the apparent motor expressions, e.g., facial expressions, that *B* has produced. In brief, appraisal theories propose to solve the problem of affective state differentiation based on a *n*-dimensional hyperplane representing the criteria of the appraisal process.

### B. Differentiable annotations of affective states

1) **Dimensional representations:** Therefore, one way of describing individuals' affective states would be to consider them as points on a plane. Based on this approach, most of the literature have used the Circumplex model to represent them, a Cartesian plane defined by two axes, namely valence – from unpleasant to pleasant – and arousal – from deactivation to activation [14]. However, this approach drew a number of criticisms for its ability to differentiate affective states; fear and anger are both defined by high arousal and negative valence, and the concept of valence itself should be rather seen as a multidimensional representation of several dimensions [15]. It has therefore been stated that more than two dimensions should be taken into account to appropriately distinguish affective states [16]. Thus, appraisal theory represents a sound framework for selecting relevant dimensions in the context of affective state annotation.

2) **Categorical representations:** Labels are another common way of approximating individual's affective states. One of the crucial points of such an approach is to select affective labels likely to be relevant in the context of interest. While basic emotion labels, i.e., fear, disgust, happiness, sadness, and anger are commonly used to model affective states [17], they are not necessarily the most appropriate in the context of healthcare and AI-patients interactions, as they refer to affect from an evolutionary perspective.

<footnote n="1">¹https://theradiacorpus.imag.fr/</footnote>
3) *Relation between categorical and dimensional representations:* Appraisal theories suggest that the use of labels represents a way for humans to characterise the conscious part of the activation states of the organism's subsystems, i.e., of the cognitive appraisal process, the physiological responses, the action tendencies, and the motor expressions [18], assuming that each affective label would represent a specific pattern of criteria of the appraisal process [13], [19].

### C. Affect corpora for healthcare

Automatic affect analysis can be applied to various aspects of healthcare [20]: to support clinicians for the diagnosis of affective disorders, to facilitate the monitoring of patients' emotions during the therapy [5], which is the focus of this study, or between therapy sessions.

Whereas numerous audiovisual corpora are available in the literature for affect recognition in human interactions [21], few of them were acquired in the context of healthcare. There are several reasons for this gap in the literature. One of them concerns the sensitive nature of the data that are protected by law and cannot be easily released. Moreover, populations related to healthcare are often fragile, and setting up experimental protocols to contrast affect corpora can be challenging because of their cognitive fatigue.

Nevertheless, there have been initiatives in the construction of affective corpora supporting clinicians during therapy. For instance, speech interactions between seniors and a virtual assistant playing the role of a health professional coach, were collected for the purpose of the EMPATHIC project [22], which aims to help the elderly maintain their independence as they age. Authors collected ~13k speech segments (~13h in total) from 136 participants over 64 years and from different countries (Spain, France, and Norway), that were further annotated by three annotators according to five labels: calm, happy, puzzled, tense, and sad, and three dimensions: arousal, valence, and dominance.

Another corpus of affect has recently been collected for children with developmental disorders using a mobile therapeutic game [23]. The corpus contains ~2k video sequences of 456 children from all over the world and was annotated by 11 individuals according to seven labels: happy, surprised, sad, fearful, angry, disgust, and neutral.

## III. DATA COLLECTION

The whole data collection protocol of this study has been documented in detail, reviewed, and approved by the Ethics Committee for Research in Grenoble Alpes with the reference: CERGA-Avis-2021-1. As the experiment lasted around three hours on average, and participants were asked to consent for the commercial exploitation of their data under the protection of the European General Data Protection Regulation laws, a 20€ gift card was awarded. Participants could ask for reimbursement of travel expenses, i.e., including public transport fares or petrol costs, for both the journey to and from the experiment site. The protocol used to collect data for the THERADIA WoZ corpus is detailed below.

<figure><img src="image_3.png" /><figcaption>Fig. 2. Office of the experimenter operating the virtual assistant. The dialog was fully scripted and overlaid onto the participant's video (A). The participant's screen (B) was only visible during the exercise completion, and the virtual assistant was visible to the participant only outside the exercise completion. The virtual assistant's rendering was consistently displayed (C) to monitor any tracking issue.</figcaption></figure>

### A. Participants

A total of 52 healthy senior participants (40 females; mean age = 67.9, SD age = 5.1) were recruited based on advertisements published in regional newspapers such as “Le Progrès” and “Le Dauphiné Libéré”. Additionnaly, 9 participants diagnosed with MCI (two females; mean age = 75.8, SD age = 2.1) were recruited via clinicians collaborating on the project. Inclusion criteria included fluency in French, either normal or corrected-to-normal vision and hearing, and informed consent prior to recordings. Regarding the level of education, 40% of healthy senior participants had a master’s degree or higher, 33% a bachelor’s degree and 27% a baccalaureate. As for MCI participants, 22% had obtained a master’s degree or higher, 22% a bachelor’s degree, and 56% a National Vocational Qualification (French CAP), or lower.

### B. CCT supported by a Virtual Assistant

All participants carried out a CCT session, which was based on eight exercises selected from the HappyNeuronPro CCT platform. The exercises involved different cognitive functions such as memory, language, attention, and planning; details on these exercises are provided in section I.A of the supplementary material.

The CCT was supported by a virtual assistant, developed by the Dynamixyz company, and controlled by one co-author of this study acting as a WoZ, cf. Figure 2. More specifically, head movements, gaze, speech, and articulation of the experimenter were captured in real-time to drive the 3D virtual assistant whose rendering was cast to the participant's screen. The speech was based on a dialogue tree designed to cover a wide range of interactions, envisioned prior to the study. If none of the planned branches made sense within a given dialogue, free intervention was allowed. Eventually, the dialogue tree was updated if the unplanned interaction was deemed relevant. Technical problems that could not be solved by the operator of the virtual assistant were addressed by a second experimenter, who was sitting outside the room for the healthy participants, and inside the room for the MCI participants.
## C. Material

Questionnaires were used to interrogate various psychological, affective and cognitive dimensions of the participants, as well as to evaluate the virtual assistant and the CCT [24]. Experiments were carried out in two quiet rooms of the EMC laboratory of Université Lyon 2, one for the participant, and one for the experimenter.

As the original lighting conditions did not allow a robust tracking of the operator of the virtual assistant, we equipped each room with an additional continuous light (Amzdeal Softbox) that provided a color temperature closed to daylight (5500K). Each room was equipped with an high-end computer, comprising a 24 inches monitor, a webcam (Logitech streamcam), a headset with a built-in unidirectional microphone (Sound blasterX H6), that was plugged to an external sound card (Presonus Audiobox), for both audio recording and streaming to the CCT application.

As pre-tests revealed random freezes in the video recordings that were not apparent during the CCT session, we used an iPhone X to provide a stable video recording on the participant's side, along with an additional audio recording captured by a far-field omnidirectional microphone. Audio data were sampled at 44.1 kHz with 16 bits, whereas video data were sampled at a constant frame rate of 30 fps with a resolution of 1920x1080 and the YUV420p color format.

## D. Procedure

Concerning the MCI participants, the session began with the completion of the consent form and the above-mentioned questionnaires. Then, after a few instructions, the experimenter left the room so that the CCT session could start. It comprised the realisation of four exercises, each performed twice. The difficulty level of the second exercise was adjusted based on the performance of the first. It was increased in the case of success and decreased in the case of difficulty. Throughout the session, participants were guided and engaged with the virtual assistant under the impression it was operated using AI technology, whereas it was actually operated by a human as a WoZ. The story line of the session started with a welcome dialogue designed to acquaint the participant with the virtual assistant, and to collect some information such as the participant's mood and motivation. This was followed by the exercise phase divided into five parts: the introductory dialogue detailing each exercise, the exercise completion, the feedback dialogue commenting on participant's performance, the repetition of the exercise, as well as the feedback dialogue, and the closing dialogue, which ended the session. Once the CCT session was terminated, the whole set-up was revealed.

Concerning the healthy senior participants, the procedure was the same, except for the following: First, a session consisted of eight exercises instead of four. Second, healthy senior participants were divided into five groups. Twenty of them were assigned to a group without induction. The special feature of this group was that it carried out two CCT sessions, one week apart. This resulted in the set-up being revealed at the end of the second session. The remaining 32 healthy senior participants were divided into four induction groups,

<table><caption>TABLE I
OVERALL STATISTICS OF THE DATA COLLECTED IN THE THERADIA-WOZ</caption><thead><tr><th>Data</th><th>Participants</th><th># Seq.</th><th>Seq. duration<br>Mean (std)</th><th>Overall<br>duration</th></tr></thead><tbody><tr><td rowspan="3">Transcribed</td><td><strong>Senior</strong></td><td>14,788</td><td>9.00 s (7.82 s)</td><td>36h59m16s</td></tr><tr><td><strong>MCI</strong></td><td>1,105</td><td>8.24 s (5.03 s)</td><td>2h31m44s</td></tr><tr><td><strong>Total</strong></td><td>15,893</td><td>8.95 s (7.66 s)</td><td>39h31m00s</td></tr><tr><td rowspan="3">Annotated</td><td><strong>Senior</strong></td><td>2,513</td><td>8.20 s (5.27 s)</td><td>5h43m27s</td></tr><tr><td><strong>MCI</strong></td><td>222</td><td>10.09 s (6.49 s)</td><td>37m20s</td></tr><tr><td><strong>Total</strong></td><td>2,735</td><td>8.35 s (5.40 s)</td><td>6h20m47s</td></tr></tbody></table>

designed to increase the chances of observing affectively-charged expressions. To this end, induction was used in one out of two of the eight exercises based on the combination of three parameters: (i) presenting the exercise as easy or hard beforehand; (ii) actually setting the difficulty as easy or hard; and (iii) giving feedback from the WoZ as critical or congratulatory. Details on the groups and the inductions are reported in section I.B of the supplementary material.

## IV. DATA SEGMENTATION AND ANNOTATION

We collected about 80 hours of data from the 52 healthy older participants, and about 6 hours of data from the 9 MCI participants. We describe in this section the methods used for the segmentation, transcription, and annotation of this data.

### A. Segmentation, transcription and selection of sequences

The segmentation of the recordings was based on the pragmatic completeness in breath groups, i.e., groups of words that are delimited by pauses taken for breathing, ensuring they formed a single proposition if they conveyed the same semantic information, or separate propositions if they related to different types of semantic information [25]. Non-verbal expressions occurring within or proximate to a proposition were marked by a diacritic, and those occurring at distance were specifically segmented. Temporal markers for proposition start and stop were both positioned at a certain distance to allow a sufficient observation space for the annotation of affect.

The transcription relied mostly on the ESLO project principles [26], respecting spelling and spoken structures; details regarding our conventions are reported in section II.A of the supplementary material. Recordings were transcribed with the ELAN software [27] by five Master students in Language Engineering at the Université Grenoble Alpes with a two-steps process: an initial stage of segmentation and transcription, followed by a subsequent verification stage conducted by another student. The segmented sequences were selected to retain only those containing expressions of affect, and a fraction was randomly removed to reduce the overall duration. Obtained sequences were then all reviewed and some were further removed to reduce redundancy. Statistics of the sequences are given in table I.
## B. Selection of affective labels and dimensions

The context of CCT aligns closely with the literature on achievement affects [28]–[30], which considers the following types of affect:

(i) activity affects generated by the task achievement itself;

(ii) outcome affects generated by the success or failure results, whether retrospective or prospective;

(iii) epistemic affects generated by a form of metacognition on the perception of the achievement;

(iv) social affects generated by another individual.

These affects are particularly relevant in the context of AI-assisted healthcare, as they comprehensively cover those that can be experienced during CCT, i.e., those generated by the CCT itself (activity affects), those generated by past or future CCT failure or success (outcome affects), those generated by the perception of the cognitive abilities involved in CCT (epistemic affects), and those generated by AI-patients interactions (social affects). Our categorical representations of affect relied on the following 23 labels: angry, annoyed, anxious, ashamed, confident, contemptuous, curious, desperate, disappointed, embarrassed, excited, frustrated, interested, guilty, happy, hopeful, impatient, proud, relaxed, sad, satisfied, surprised, and upset.

The dimensions consisted of the four most important appraisal dimensions in the context of affect recognition [13], [19], [31], [32], namely: novelty, intrinsic pleasantness, goal conduciveness, and coping, plus arousal. In the context of appraisal theory, arousal is a marker of the depth of the appraisal process, which is relevant to qualify the intensity of the affective state.

## C. Annotation tool and guidelines

We took benefit from previous collaborations with the company Viadialog<fnref n="2" /> engaged in the development of a web-based annotation tool originally tailored for audio data: ANNOT. This tool has been adapted to the needs of our project and is available online<fnref n="3" />. The platform allowed annotators – here 6 French students (3 females) from different backgrounds (engineering, business, and biology) and places (Lyon and Toulouse) – to view and annotate the selected audiovisual sequences at home, using their own computer.

The annotation consisted of three steps: a time-continuous annotation, cf. Figure 3, followed by a summary value, for each of the five dimensions, and a summary value of intensity for one or multiple labels. Thus, each sequence was viewed five times, once for each dimension. At the end of the time-continuous annotation, a graph showing the dynamics of the dimension over time was presented to the annotators, who could then validate or perform the annotation again. After the validation, annotators were asked to provide a summary value for the whole sequence. Finally, once all dimensions were annotated, annotators were asked to indicate if one or more labels could describe the affect they had recognised in the sequence and provide an intensity value for them. Annotators

<figure><img src="image_4.png" /><figcaption>Fig. 3. Example of the ANNOT web-base platform for time-continuous annotation of the intrinsic pleasantness dimension.</figcaption></figure>

could annotate as many labels as desired, or none at all if the sequence lacked any particular affect. Further details on the annotations steps are reported in section II.C of the supplementary material.

Before starting, annotators participated in a training session with the project investigators, including an expert in appraisal theories of emotion. The purpose of this session was to familiarise the annotators with the concepts of appraisal theories, i.e., the meaning of dimensions and affective labels, and to explain in detail the guidelines for the annotation process. Annotators were instructed to put themselves in the participant's shoes to assess the emotional expression conveyed in the selected sequences, considering factors such as the chosen words, their oral delivery, and the expressive dynamics of the face and body. It was emphasised that there were no right or wrong answers, as the task was inherently subjective. Additionally, they were instructed to perform the task in full-screen mode, in a quiet environment, with sessions lasting no longer than one hour, followed by ten-minute breaks. The annotation process was continuously monitored through an online tracking table, updated daily by the annotators, with additional communication via email or phone as needed. After the training session, each annotator received a document summarising all these points, including definitions of dimensions and labels. Each dimension was defined as follows:

* **Novelty:** "To what extent does the participant feel that what is happening is predictable or unexpected?"; ranging from *predictable* to *unexpected*.

* **Intrinsic pleasantness:** "At what point does the participant express pleasure or displeasure?"; ranging from *unpleasantness* to *pleasantness*.

* **Goal conduciveness:** "How much does what is happening seems to correspond to the participant's wishes?"; ranging from *not at all* to *absolutely*.

* **Coping:** "How well does the participant seem able to cope with what is happening?"; ranging from *not at all* to *absolutely*.

* **Arousal:** "How asleep or awake does the participant seem?"; ranging from *asleep* to *awake*.

<footnote n="2">²https://viadialog.com/</footnote>

<footnote n="3">³https://github.com/fabienringeval/ANNOT/</footnote>
<figure><img src="image_5.png" /><figcaption>Fig. 4. Frequency (logarithmic scale) of the annotated labels according to the agreement of at least one, two or three annotators among six.</figcaption></figure>

V. CORPUS ANALYSIS

This section provides a comprehensive statistical analysis of the annotation data. We first assessed the annotation frequency of the affective labels as well as the inter-annotator agreement on their intensity. We then evaluated the cognitive delay involved in the annotation of each dimension, as well as the inter-annotator agreement. Relationships between dimensions and affective labels were also estimated, and a label core set was finally selected, based on compliance with consistency criteria relating to each of the analyses mentioned above.

A. Categorical annotations

The frequency at which the affective labels were selected by at least one, two, or three annotators is reported in Figure 4. Based on the sample of annotation frequency per label for at least one annotator, Cohen's *d* was calculated to identify the theoretical value at which the effect size of its deviation from the sample mean can be considered at least "small", i.e., Cohen's *d* > 0.2 [33]; this theoretical value was equal to 747 sequences. The annotation frequency per label for at least one annotator is reported in Table II, and shown in bold when above this theoretical value.

We evaluated the inter-annotator agreement on the presence of each affect label, i.e., when the intensity of a label was superior to zero, for each pair of annotators, and averaged over all pairs, by computing the Unweighted Average Recall (UAR) as follows:

$$
UAR = \frac{1}{k} \sum_{i=1}^{k} \frac{N_c^i}{N^i} \qquad (1)
$$

where *k* is the total number of classes, which corresponds here to the presence or absence of a label. *N*<sub>*c*</sub><sup>*i*</sup> is the total number of annotated sequences for a given affect label, and *N*<sub>*c*</sub><sup>*i*</sup> is the number of commonly identified classes, whether present or absent. Results are reported in Table II. According to a two-tailed *z*-test, the inter-annotator agreement would be defined

<table><caption>TABLE II
STATISTICS ON THE ANNOTATED AFFECTIVE LABELS: FREQUENCY, INTER-ANNOTATOR AGREEMENT ON THE PRESENCE, AND INTENSITY, AND CORRELATION TO APPRAISAL'S DIMENSIONS. MOST RELEVANT LABELS IN THE CONTEXT OF AI-ASSISTED CCT ARE MARKED IN BOLD</caption><thead><tr><th rowspan="2">Label</th><th rowspan="2">Frequency</th><th colspan="2">Agreement on</th><th rowspan="2">Cor. to Appraisals</th></tr><tr><th>Presence</th><th>Intensity</th></tr></thead><tbody><tr><td><strong>Relaxed</strong></td><td><strong>1683</strong></td><td><strong>59.24 %</strong></td><td><strong>.114</strong></td><td>✓</td></tr><tr><td><strong>Interested</strong></td><td><strong>1525</strong></td><td><strong>54.99 %</strong></td><td><strong>.194</strong></td><td>✓</td></tr><tr><td><strong>Frustrated</strong></td><td><strong>1315</strong></td><td><strong>59.05 %</strong></td><td><strong>.211</strong></td><td>✓</td></tr><tr><td><strong>Satisfied</strong></td><td><strong>1216</strong></td><td><strong>60.69 %</strong></td><td><strong>.097</strong></td><td>✓</td></tr><tr><td><strong>Confident</strong></td><td><strong>1238</strong></td><td><strong>60.97 %</strong></td><td><strong>.116</strong></td><td>✓</td></tr><tr><td><strong>Happy</strong></td><td><strong>902</strong></td><td><strong>62.81 %</strong></td><td><strong>.280</strong></td><td>✓</td></tr><tr><td><strong>Surprised</strong></td><td><strong>851</strong></td><td><strong>58.44 %</strong></td><td><strong>.260</strong></td><td>✓</td></tr><tr><td><strong>Annoyed</strong></td><td><strong>856</strong></td><td><strong>57.52 %</strong></td><td><strong>.130</strong></td><td>✓</td></tr><tr><td><strong>Desperate</strong></td><td><strong>826</strong></td><td><strong>58.99 %</strong></td><td><strong>.276</strong></td><td>✓</td></tr><tr><td><strong>Anxious</strong></td><td><strong>783</strong></td><td><strong>54.53 %</strong></td><td><strong>.283</strong></td><td>✓</td></tr><tr><td>Embarrassed</td><td>745</td><td>51.43 %</td><td>-.256</td><td></td></tr><tr><td>Disappointed</td><td>520</td><td><strong>56.00 %</strong></td><td><strong>.387</strong></td><td></td></tr><tr><td>Guilty</td><td>400</td><td><strong>57.44 %</strong></td><td><strong>.129</strong></td><td>✓</td></tr><tr><td>Contemptuous</td><td>363</td><td><strong>53.61 %</strong></td><td><strong>.359</strong></td><td></td></tr><tr><td>Angry</td><td>311</td><td><strong>56.76 %</strong></td><td><strong>.487</strong></td><td>✓</td></tr><tr><td>Curious</td><td>328</td><td><strong>55.60 %</strong></td><td><strong>.439</strong></td><td></td></tr><tr><td>Hopeful</td><td>223</td><td><strong>55.62 %</strong></td><td><strong>.480</strong></td><td>✓</td></tr><tr><td>Proud</td><td>249</td><td><strong>58.79 %</strong></td><td>.015</td><td></td></tr><tr><td>Excited</td><td>159</td><td><strong>56.39 %</strong></td><td><strong>.150</strong></td><td></td></tr><tr><td>Ashamed</td><td>144</td><td><strong>54.22 %</strong></td><td>-.014</td><td></td></tr><tr><td>Impatient</td><td>138</td><td><strong>53.14 %</strong></td><td>-.338</td><td></td></tr><tr><td>Upset</td><td>74</td><td><strong>53.48 %</strong></td><td><strong>1.00</strong></td><td></td></tr><tr><td>Sad</td><td>63</td><td><strong>57.28 %</strong></td><td><strong>.668</strong></td><td></td></tr></tbody></table>

as statistically above chance if it would exceed 53%, which is
the case for all the labels excepted embarrassed.

We further evaluated the inter-rater agreement on the in-
tensity of the affective labels using the Pearson's Correlation
Coefficient (PCC). The inter-rater agreement was defined as
sufficiently high if its Cohen's *d*, i.e., its PCC value divided
by the sample standard deviation, is superior to .2 [33], which
corresponds to a PCC greater than .065. This concerned most
of the affective labels, which are reported in bold style.
<caption>TABLE III</caption>

INTER-ANNOTATOR AGREEMENT ON EACH SEQUENCE, OR CCT SESSION, COMPUTED WITH THE PEARSON'S CORRELATION COEFFICIENT FOR BOTH TIME-CONTINUOUS AND SUMMARY VALUES OF APPRAISAL DIMENSIONS; MEAN (STANDARD-DEVIATION)

<table><thead><tr><th></th><th>Sequence</th><th>Session</th></tr></thead><tbody><tr><td>Time-continuous values</td><td></td><td></td></tr><tr><td>Arousal</td><td>.381 (.034)</td><td>.425 (.060)</td></tr><tr><td>Coping</td><td>.586 (.026)</td><td>.573 (.032)</td></tr><tr><td>Goal conduciveness</td><td>.602 (.019)</td><td>.539 (.031)</td></tr><tr><td>Intrinsic pleasantness</td><td>.588 (.088)</td><td>.547 (.038)</td></tr><tr><td>Novelty</td><td>.307 (.088)</td><td>.294 (.128)</td></tr><tr><td>Summary values</td><td></td><td></td></tr><tr><td>Arousal</td><td>.248 (.039)</td><td>.228 (.065)</td></tr><tr><td>Coping</td><td>.679 (.024)</td><td>.676 (.038)</td></tr><tr><td>Goal conduciveness</td><td>.695 (.015)</td><td>.680 (.032)</td></tr><tr><td>Intrinsic pleasantness</td><td>.662 (.020)</td><td>.619 (.036)</td></tr><tr><td>Novelty</td><td>.401 (.113)</td><td>.403 (.150)</td></tr></tbody></table>

<figure><img src="image_6.png" /><figcaption>Fig. 5. Heatmap of the PCC between affective labels and summary values of appraisal dimensions aggregated across all annotators. The sign (+) or (-) after each label denotes whether it is positive or negative, respectively. Dimensions are as follows: AR: arousal, CP: coping, GC: goal conduciveness, IP: intrinsic pleasantness, NV: novelty. The correlations shown represent the average between annotators for whom the correlation was significant ($p < 0.05$) after Bonferroni correction and of identical direction. Positive or negative direction is shown in blue or red, respectively. A correlation present for a single annotator was considered null.</figcaption></figure>

## B. Dimensional annotations

The cognitive delay in the continuous annotation of each dimension was evaluated, and results showed that goal conduciveness (3.3 seconds) had a significantly longer delay than the other dimensions (less than 2 seconds), suggesting that annotators had more difficulty assessing it. This evaluation is detailed in section III.A of the supplementary material.

We assessed the inter-annotator agreement on the dimensions for both time-continuous and summary annotations, using the PCC, which was computed either on each sequence, or each CCT session, cf. Table III. An high agreement was observed on the dimensions of coping, goal conduciveness, and intrinsic pleasantness, with a more pronounced consistency observed in summary annotations compared to time-continuous annotations, which is expected since time-continuous data imply a greater variation in the possible outcomes. A lower agreement was however found for arousal and novelty, which might be explained by several reasons as detailed below.

The novelty dimension was assumed to be more difficult to annotate, as it probably concerned very specific cases of our study, e.g., unplanned feedback from the virtual assistant on the participant's performance, occurring in fewer sequences than the other dimensions. This assumption was corroborated by an analysis evidencing that the frequency of high-value annotations was lower for novelty compared to other dimensions; for a high-value threshold set at .85, the proportion of high-value annotations for novelty accounted for only 0.13% of the time-continuous annotation data, whereas other dimensions were at least twice as frequent: 0.30% for arousal, 0.24% for intrinsic pleasantness, 0.62% for goal conduciveness, and 0.36% for coping. A similar trend was observed in summary annotations, with the occurrence of high-value annotations for novelty accounting to 0.26%, whereas all other dimensions were twice as prevalent; arousal: 0.92%, intrinsic pleasantness: 0.52%, goal conduciveness: 1.35%, and coping: 2.16%.

The arousal dimension was thought to be more difficult to annotate, as the context of the study, i.e. exercising on a computer and conversing with a virtual assistant, is likely to provide little variation for this dimension. This assumption is supported by the fact that the variance of arousal dimension was systematically among the lowest, which was observed for both time-continuous annotations – arousal: .010, novelty: .022, intrinsic pleasantness: .009, goal conduciveness: .017, coping: .020 –, and summary annotations – arousal: .008, novelty: .026, intrinsic pleasantness: .012, goal conduciveness: .021, coping: .027.

## C. Relation between dimensions and affective labels

Further analysis of annotator agreement was conducted by exploring the relationship between summary values of appraisal dimensions and affective labels of each annotator separately. Specifically, the PCC was calculated between each pair of appraisal dimensions and affective labels for all sequences in which the value of the label concerned differed from zero; cf. section III.B of the supplementary material for a comprehensive view of the heatmap of each annotator. A correlation was considered consistent if it was statistically significant and in the same direction for at least two annotators. The average of consistent correlations between annotators are reported in Figure 5. The labels that are consistently correlated with at least one appraisal dimension are marked with "✓" in the Table II.

Broadly speaking, our results are in line with the predictions of appraisal theories regarding the link between affective labels and dimensions: positive affective labels are correlated with the ability to cope, goal conduciveness, and pleasantness, while negative affective labels are correlated with disability to cope, goal obstructiveness, and unpleasantness.
<figure><img src="image_7.png" /><figcaption>Fig. 6. Overview of the predictive experiments performed on the corpus.</figcaption></figure>

Some of our results are, however, unexpected. The anger affective label is correlated with an absence of coping, whereas we would expect the opposite [34], as it generally refers to a situation in which an event is appraised as goal obstructive, but individuals appraise themselves capable of coping with it, e.g., being mugged in the street but wanting to defend themselves. In the context of CCT, where participants sit in front of a computer, different forms of anger may likely occur, probably involving an inability to cope. As expected, the affective label of surprise is highly correlated with novelty. Even though the surprise affective label is generally considered to be positive or negative depending on the context, it seems to be rather negative in our results as it is correlated with goal obstructiveness, and no coping. This could be explained by the fact that the occurrence of unexpected situations were more related to negative events such as unexpected negative feedback from the virtual assistant.

## D. Label core set selection

A core set of the likely most relevant affective labels in the context of AI-assisted CCT was defined based on our analyses. Four main criteria of selection were used: (i) the frequency is significantly higher than the average on all affective labels; (ii) the agreement on presence is significantly higher than chance; (iii) the agreement on intensity is sufficiently high; and (iv) there is a significant correlation, in the same direction, with at least one dimension for at least two annotators. Ten affective labels meet these four criteria, five positive and five negative, cf. Table II.

## VI. AUTOMATIC RECOGNITION OF AFFECTIVE STATES

We report on automatic prediction of affective labels and dimensions using audio, textual, and video data, cf. overview in Figure 6. The dataset was partitioned into training (56%), validation (23%), and test sets (21%), carefully balanced regarding gender, health status, affect induction conditions, education levels, and distribution of affect annotations, cf. Table IV and section IV of supplementary material.

### A. Representations of audio, textual and video data

Expert and deep representations were extracted from all modalities<fnref n="4" />. Expert features rely on the statistical description of perceptually salient patterns in the data, and involve

<table><caption>TABLE IV
SUMMARY STATISTICS OF THE PARTITIONS</caption><thead><tr><th></th><th>Training</th><th>Val</th><th>Test</th><th>Total</th></tr></thead><tbody><tr><td>Participants</td><td>34</td><td>14</td><td>13</td><td>61</td></tr><tr><td>Videos</td><td>1110</td><td>851</td><td>774</td><td>2735</td></tr><tr><td>Duration</td><td>2h44m36s</td><td>1h51m42s</td><td>1h44m28s</td><td>6h20m47s</td></tr><tr><td colspan="5">Population group</td></tr><tr><td>Senior</td><td>29</td><td>12</td><td>11</td><td>52</td></tr><tr><td>MCI</td><td>5</td><td>2</td><td>2</td><td>9</td></tr><tr><td colspan="5">Gender (seniors)</td></tr><tr><td>Male</td><td>6</td><td>3</td><td>3</td><td>12</td></tr><tr><td>Female</td><td>23</td><td>9</td><td>8</td><td>40</td></tr><tr><td colspan="5">Emotion Induction (seniors)</td></tr><tr><td>With induction</td><td>18</td><td>7</td><td>7</td><td>32</td></tr><tr><td>Without induction</td><td>11</td><td>5</td><td>4</td><td>20</td></tr><tr><td colspan="5">Education (seniors)</td></tr><tr><td>High school</td><td>5</td><td>2</td><td>1</td><td>8</td></tr><tr><td>Professional certificate</td><td>3</td><td>1</td><td>2</td><td>6</td></tr><tr><td>Undergraduate</td><td>9</td><td>5</td><td>3</td><td>17</td></tr><tr><td>Graduate</td><td>12</td><td>4</td><td>5</td><td>21</td></tr></tbody></table>

relatively light-weight computational resources, whereas deep representations exploit self-supervised learning (SSL) methods on large scale datasets [35].

1) *Expert based representations:* We used the Mel-scale Filter Banks (MFBs) [36], Term Frequency - Inverse Document Frequency (TF-IDF) [37], and Facial Action Units (FAUs) [38], as representations of the audio, textual, and visual data, respectively.

MFBs are popular features in acoustic processing. They are based on a simple signal processing pipeline which consists in taking the Fourier transform of the acoustic signal through a sliding window (25 ms shifted forward in time every 10 ms), and then mapping the power spectrum to a mel-scale (80 filter banks), which is a non-linear perceptual scale based on the human auditory system. MFBs were extracted using the SpeechBrain toolkit [39].

The TF-IDF is a classical feature in text processing, capturing the frequency of a word within a document (TF), relative to its rarity across a given set of documents (IDF). The method therefore tokenises the words of a sentence, which is represented by a vector of the count of the words it contains, and further normalised by the term frequency across the entire training partition of the dataset.

FAUs describe the intensity of movements of facial muscles related to an apparent facial expression, and are commonly used as facial descriptors in affective computing. FAUs were extracted using OpenFace [40], which provides estimations of the intensity of a set of 17 FAUs with a framerate of 30 Hz.

2) *Self-supervised representations:* Deep representations of audio, textual, and video data include the following models: Wav2Vec2 [41], BERT [42], and CLIP [43], respectively.

We used a multilingual Wav2Vec2 model that exploits both convolutional and transformer layers to distinguish positive and negative samples of speech in a contrastive approach. The model<fnref n="5" /> we used has a large architecture, and was trained on 56

<footnote n="4">⁴In order to make the comparison fair, textual data did not rely on the human transcriptions but on the outputs of an ASR, which is evaluated in detail in section IV.A of the supplementary material.</footnote>

<footnote n="5">⁵https://huggingface.co/voidful/wav2vec2-xlsr-multilingual-56</footnote>
different languages [44], including French, and then fine-tuned
for ASR.

Transformer-based representations such as BERT are pop-
ular in text processing, as they can encode the context of
the words, which is crucial for affect related tasks [45], [46].
We used a BERT model<fnref n="6" /> fine-tuned for sentiment analysis on
six languages (English, Dutch, German, French, Spanish, and
Italian), and with the base architecture.

CLIP<sup>7</sup> is a model developed by OpenAI [43] and trained on a large dataset that includes both images and their corresponding textual descriptions allowing the model to learn the semantic relationship between visual and textual information. The architecture consists of image and text encoders that encode the input image and text into fixed-length embeddings using a contrastive learning mechanism.

B. *Models and training strategies*

Three predictive models were evaluated: Linear, Multi-
Layer Perceptron (MLP), and Gated Recurrent Units (GRU).
Outputs of the Linear and MLP models were averaged over the
sequence, whereas the last output of the GRU model was used
to represent the output of the whole sequence. Preliminary
results obtained in the prediction of the affective labels showed
that the performance of the MLP model was consistently better
than the performance of Linear and GRU models, both for the
classification and regression tasks.

For dimension summaries, the model was trained to predict
the five dimensions altogether in a multi-task approach. Our
first experiments revealed that the Linear model consistently
outperformed both the MLP and GRU models.

To predict the time-continuous annotations of the dimen-
sions, it was necessary to resample certain features to align
with the annotation frame rate of 10 Hz. Audio representations,
sampled at 100 Hz for MFBs, and 50 Hz for Wav2vec2, were
averaged into 10 ms chunks to match the annotation frequency.
The same strategy was applied to the FAUs and CLIP visual
representations. For textual representations, we utilised BERT
exclusively, as TF-IDF provides a single feature vector for
the entire sequence. Textual features were aligned to the
sequence length using spline interpolation, ensuring a smooth
and continuous representation of the features over time.

Since the Linear and MLP models did not perform well in
predicting time-continuous dimensions, we explored various
GRU model architectures:

1) One-Layer GRU: A single-layered GRU with a hidden size set to either half the input feature size or equal to the input feature size.

2) Two-Layers GRU: Similar to the One-Layer GRU but with two hidden layers.

3) Three-Layers GRU: Similar to before, but with three hidden layers.

Additionally, we tested a fixed set of hidden sizes across all
GRU models: 128, 256, and 512. The results indicated that a
GRU model with three hidden layers, each matching the input

size, performed optimally for deep representations. In contrast,
the model with a hidden size of 256 produced the best results
for hand-crafted representations.

For multimodal fusion, we explored both middle and late
fusion techniques to address the issue of asynchronous inputs.
The best performing approach for both labels and dimension
summaries was decision-based fusion using an Ordinary Least
Squares (OLS) regression model<fnref n="8" />. For time-continuous dimen-
sions, fusion was achieved through a GRU functioning as a
weighted averaging layer, effectively capturing the temporal
dynamics of the features. The GRU weights were optimised
on the training set.

All the models were trained with the Adam optimizer and an
initial learning rate of 10⁻³ for affective labels and 10⁻⁴ for
dimensions. The batch size and gradient accumulation were set
to 10 for labels and 1 for dimensions. The maximum number
of training epochs was set to 50 and an early stopping strategy
was used with five epochs.

The models were trained using the Mean Squared Error (MSE) as the loss function, and for evaluation, the Concordance Correlation Coefficient (CCC) was calculated on all sequences [47]. All experiments were carried out using Pytorch<fnref n="9" /> [48], with random seeds set to zero. The computer's operating system was Debian GNU/Linux 10, and the GPU used to train the models was an NVIDIA Quadro RTX 6000 with 23 gigabytes of memory, CUDA version 11.3.

C. *Affect prediction for the core set labels*

The results for label intensity predictions, presented in Table V, reveal some strong dependencies between modalities and affective labels; see section IV.B of the supplementary material for additional details on prediction biases related to population demographics. For instance, the label “happy” was most accurately assessed via the video modality, “interested” through the textual modality, and “frustrated” and “surprised” using the audio modality with deep representations. The choice between expert-crafted and deep features also introduced notable differences in the results according to the modality. For example, the label “frustrated” achieved a significantly higher CCC (.327) with TF-IDF features compared to MFB (.151), whereas the reverse trend was observed when using SSL representations. Overall, deep representations across textual, audio, and video modalities consistently outperformed their hand-crafted counterparts on average.

The fusion of the different modalities yielded the best results
on average and for most affective labels. An analysis of the
fusion weights revealed that the importance of each modality
varied significantly based on its representation. For instance,
when using expert features, TF-IDF had a substantially greater
influence on the results, with an averaged fusion coefficient
of 0.8, compared to 0.1 for audio (MFBs) and video (FAUs).
However, with deep representations, the contributions of audio
and video surpassed that of text, with average coefficients of
.28, .35, and .35 for text, audio, and video respectively.

<footnote n="6">⁶https://huggingface.co/nlptown/bert-base-multilingual-uncased-sentiment</footnote>

<footnote n="7">⁷https://openai.com/research/clip</footnote>

<footnote n="8">⁸https://scikit-learn.org</footnote>

<footnote n="9">⁹https://pytorch.org</footnote>
<table><caption>TABLE V
CORE-SET LABEL INTENSITY PREDICTION RESULTS (CCC)</caption><thead><tr><th rowspan="2">Label</th><th colspan="4">Expert Representations</th><th colspan="4">Deep Representations</th></tr><tr><th>Text TF-IDF</th><th>Audio MFB</th><th>Video FAU</th><th>Multi.</th><th>Text BERT</th><th>Audio W2V2</th><th>Video CLIP</th><th>Multi.</th></tr></thead><tbody><tr><td>Annoyed</td><td>.075</td><td>.047</td><td>.274</td><td>.255</td><td>.255</td><td>.231</td><td>.311</td><td>.280</td></tr><tr><td>Anxious</td><td>.115</td><td>.081</td><td>.129</td><td>.171</td><td>.087</td><td>.192</td><td>.123</td><td>.214</td></tr><tr><td>Confident</td><td>.233</td><td>.176</td><td>.168</td><td>.422</td><td>.473</td><td>.482</td><td>.247</td><td>.487</td></tr><tr><td>Desperate</td><td>.172</td><td>.148</td><td>.193</td><td>.259</td><td>.221</td><td>.277</td><td>.175</td><td>.252</td></tr><tr><td>Frustrated</td><td>.327</td><td>.151</td><td>.175</td><td>.387</td><td>.222</td><td>.340</td><td>.239</td><td>.465</td></tr><tr><td>Happy</td><td>.117</td><td>.273</td><td>.446</td><td>.490</td><td>.108</td><td>.303</td><td>.462</td><td>.508</td></tr><tr><td>Interested</td><td>.147</td><td>.135</td><td>-.023</td><td>.207</td><td>.364</td><td>.204</td><td>.133</td><td>.300</td></tr><tr><td>Relaxed</td><td>.314</td><td>.300</td><td>.385</td><td>.515</td><td>.363</td><td>.245</td><td>.440</td><td>.529</td></tr><tr><td>Satisfied</td><td>.323</td><td>.225</td><td>.253</td><td>.428</td><td>.057</td><td>.287</td><td>.326</td><td>.486</td></tr><tr><td>Surprised</td><td>.167</td><td>.165</td><td>.024</td><td>.197</td><td>.143</td><td>.238</td><td>.095</td><td>.275</td></tr><tr><td>Average</td><td>.199</td><td>.170</td><td>.202</td><td>.333</td><td>.229</td><td>.280</td><td>.255</td><td>.380</td></tr></tbody></table>

<table><caption>TABLE VI
CORE-SET LABEL CLASSIFICATION RESULTS (%UAR)</caption><thead><tr><th rowspan="2">Label</th><th colspan="4">Expert Representations</th><th colspan="4">Deep Representations</th></tr><tr><th>Text TF-IDF</th><th>Audio MFB</th><th>Video FAU</th><th>Multi.</th><th>Text BERT</th><th>Audio W2V2</th><th>Video CLIP</th><th>Multi.</th></tr></thead><tbody><tr><td>Annoyed</td><td>63.2</td><td>57.5</td><td>62.4</td><td>66.0</td><td>63.1</td><td>63.7</td><td>65.1</td><td>69.4</td></tr><tr><td>Anxious</td><td>66.3</td><td>59.3</td><td>57.0</td><td>67.2</td><td>65.5</td><td>64.9</td><td>58.2</td><td>67.6</td></tr><tr><td>Confident</td><td>70.4</td><td>66.3</td><td>59.8</td><td>71.6</td><td>73.4</td><td>73.1</td><td>59.5</td><td>76.0</td></tr><tr><td>Desperate</td><td>71.3</td><td>68.6</td><td>62.4</td><td>72.1</td><td>69.3</td><td>74.9</td><td>65.6</td><td>75.7</td></tr><tr><td>Frustrated</td><td>72.2</td><td>63.9</td><td>63.3</td><td>74.1</td><td>70.4</td><td>69.9</td><td>63.4</td><td>74.5</td></tr><tr><td>Happy</td><td>65.4</td><td>62.5</td><td>71.1</td><td>73.5</td><td>65.8</td><td>66.8</td><td>73.9</td><td>76.7</td></tr><tr><td>Interested</td><td>61.7</td><td>57.5</td><td>50.1</td><td>62.4</td><td>64.9</td><td>60.5</td><td>59.1</td><td>64.1</td></tr><tr><td>Relaxed</td><td>68.7</td><td>64.9</td><td>68.7</td><td>75.1</td><td>69.0</td><td>67.3</td><td>64.1</td><td>74.0</td></tr><tr><td>Satisfied</td><td>70.4</td><td>63.9</td><td>64.0</td><td>72.2</td><td>73.9</td><td>70.0</td><td>63.8</td><td>75.3</td></tr><tr><td>Surprised</td><td>62.3</td><td>56.8</td><td>52.2</td><td>61.8</td><td>62.1</td><td>61.7</td><td>54.1</td><td>62.3</td></tr><tr><td>Average</td><td>67.2</td><td>62.1</td><td>61.1</td><td>69.6</td><td>67.7</td><td>67.3</td><td>62.7</td><td>71.6</td></tr></tbody></table>

An affective label was considered present in a given sequence if at least one annotator reported an intensity value for that label. To predict these labels, we explored two modeling approaches: training a binary classification model for each affective label or deriving a threshold from the regression models. Since the binary classification approach yielded significantly poorer results compared to the thresholded-based decision from regression models, we opted for the latter in our experiments.

The results, presented in Table VI, indicate some dependencies between the best-performing modality and the affective labels, although these differences are less pronounced than in previous experiments. Overall, no significant performance gap was observed between expert and deep representations across modalities, with the exception of the audio modality. Additionally, fusion consistently enhanced performance across most scenarios.

## D. Dimensions summaries prediction

Results are presented in Table VII for two different gold standards: the average of annotations and the median; see section IV.B of the supplementary material for further details on prediction bias related to population demographics. Comparing these gold standards, the average of annotations consistently yielded the best performance across all five dimensions. In terms of individual dimensions, arousal was best predicted by the audio modality, with no significant improvement observed from fusion with the other modalities. It also had the lowest scores among the dimensions, which might be explained by the factors discussed in Section V-C.

<table><caption>TABLE VII
DIMENSION SUMMARY PREDICTION (CCC)</caption><thead><tr><th rowspan="2">Dimensions</th><th rowspan="2"></th><th colspan="7">Modalities</th></tr><tr><th>Text</th><th>Audio</th><th>Video</th><th>Multimodal</th></tr></thead><tbody><tr><td rowspan="2">Arousal</td><td>Mean</td><td>.311</td><td>.195</td><td>.390</td><td>.309</td><td>.234</td><td>.209</td><td>.356</td><td>.310</td></tr><tr><td>Median</td><td>.264</td><td>.178</td><td>.276</td><td>.282</td><td>.252</td><td>.219</td><td>.326</td><td>.276</td></tr><tr><td rowspan="2">Coping</td><td>Mean</td><td>.506</td><td>.452</td><td>.294</td><td>.499</td><td>.336</td><td>.460</td><td>.599</td><td>.684</td></tr><tr><td>Median</td><td>.488</td><td>.459</td><td>.302</td><td>.493</td><td>.329</td><td>.497</td><td>.603</td><td>.666</td></tr><tr><td rowspan="2">Goal conduciveness</td><td>Mean</td><td>.520</td><td>.495</td><td>.347</td><td>.428</td><td>.345</td><td>.380</td><td>.630</td><td>.662</td></tr><tr><td>Median</td><td>.491</td><td>.468</td><td>.291</td><td>.426</td><td>.361</td><td>.389</td><td>.605</td><td>.642</td></tr><tr><td rowspan="2">Intrinsic pleasantness</td><td>Mean</td><td>.399</td><td>.313</td><td>.296</td><td>.357</td><td>.500</td><td>.573</td><td>.606</td><td>.609</td></tr><tr><td>Median</td><td>.376</td><td>.202</td><td>.261</td><td>.346</td><td>.483</td><td>.463</td><td>.574</td><td>.594</td></tr><tr><td rowspan="2">Novelty</td><td>Mean</td><td>.429</td><td>.377</td><td>.357</td><td>.371</td><td>.015</td><td>.312</td><td>.465</td><td>.559</td></tr><tr><td>Median</td><td>.329</td><td>.265</td><td>.277</td><td>.298</td><td>.021</td><td>.172</td><td>.376</td><td>.420</td></tr></tbody></table>

Coping was the most accurately predicted dimension, with all modalities performing similarly well and demonstrating complementary effects during fusion. Goal conduciveness was best predicted by textual representations, with fusion incorporating audio and video modalities further improving performance. Intrinsic pleasantness was most effectively captured by the video modality, with textual representations proving relevant only when computed over the entire sequence using TF-IDF. Deep audio representations also performed comparably well. Finally, novelty was particularly challenging for the video modality, which yielded extremely low scores, while textual representations proved to be more effective in capturing this dimension.

By analysing the relative importance of each modality based on the coefficients from the linear regression model, we found that audio features contributed very little to predicting arousal, especially when using expert features. This could explain the drop in performance observed with the multimodal approach, compared to using the audio modality alone. For the others dimensions, the contributions of each modality were influenced by the complexity of the representations. When using deep features, the audio modality emerged as the most influential. However, this contribution was diminished when expert features were used, with textual information playing a more significant role.

## E. Continuous dimensions prediction

Results are presented in Table VIII for three gold standards: the average and the median of the time-continuous annotations, and a version based on the Evaluator Weighted Estimator (EWE), which calculates a weighted mean of the annotations using the average cross-CCC between each rater's scores as weights [49]; for further details on prediction bias based on population demographics, see section IV.B of the supplementary material. Overall, the EWE approach consistently outperformed the others two gold standards across the five dimensions.
<table><caption>TABLE VIII
TIME-CONTINUOUS DIMENSION PREDICTION (CCC)</caption><thead><tr><th rowspan="2">Dimensions</th><th rowspan="2"></th><th colspan="5">Modalities</th></tr><tr><th>Text</th><th>Audio</th><th>Video</th><th>Multimodal</th></tr></thead><tbody><tr><td rowspan="3">Arousal</td><td>Mean</td><td>.533</td><td>.575</td><td>.545</td><td>.539</td><td>.500</td><td>.545</td></tr><tr><td>Median</td><td>.644</td><td>.705</td><td>.665</td><td>.696</td><td>.643</td><td>.684</td></tr><tr><td>EWE</td><td>.554</td><td>.709</td><td>.699</td><td>.651</td><td>.643</td><td>.691</td></tr><tr><td rowspan="3">Coping</td><td>Mean</td><td>.356</td><td>.254</td><td>.281</td><td>.198</td><td>.239</td><td>.346</td></tr><tr><td>Median</td><td>.186</td><td>.269</td><td>.406</td><td>.213</td><td>.260</td><td>.340</td></tr><tr><td>EWE</td><td>.376</td><td>.308</td><td>.422</td><td>.183</td><td>.251</td><td>.418</td></tr><tr><td rowspan="3">Goal conduciveness</td><td>Mean</td><td>.320</td><td>.230</td><td>.236</td><td>.207</td><td>.199</td><td>.357</td></tr><tr><td>Median</td><td>.168</td><td>.224</td><td>.390</td><td>.218</td><td>.249</td><td>.347</td></tr><tr><td>EWE</td><td>.310</td><td>.248</td><td>.339</td><td>.191</td><td>.183</td><td>.420</td></tr><tr><td rowspan="3">Intrinsic pleasantness</td><td>Mean</td><td>.258</td><td>.202</td><td>.188</td><td>.262</td><td>.231</td><td>.317</td></tr><tr><td>Median</td><td>.125</td><td>.195</td><td>.308</td><td>.272</td><td>.256</td><td>.324</td></tr><tr><td>EWE</td><td>.262</td><td>.250</td><td>.289</td><td>.242</td><td>.245</td><td>.370</td></tr><tr><td rowspan="3">Novelty</td><td>Mean</td><td>.224</td><td>.154</td><td>.196</td><td>.094</td><td>.175</td><td>.231</td></tr><tr><td>Median</td><td>.139</td><td>.116</td><td>.308</td><td>.111</td><td>.203</td><td>.274</td></tr><tr><td>EWE</td><td>.262</td><td>.203</td><td>.279</td><td>.071</td><td>.202</td><td>.339</td></tr></tbody></table>

Arousal was the best predicted dimension, but this was primarily due to the low variance observed within this dimension, as explained in the previous section. For coping, goal conduciveness and novelty, Wav2vec2 audio representations outperformed textual features, which contrasts with the findings on the summarised dimensions. This discrepancy may be attributed to the fact that textual features provide a static representation, which may not capture the continuous variations within the data. Regarding intrinsic pleasantness, the three modalities performed similarly and proved complementary during fusion, which is consistent with the previous section's findings.

Additionally, the multimodal approach consistently improved performance. An analysis of the fusion weights revealed that when using the mean of annotations as the gold standard, the model evenly distributed importance across all modalities. However, when using the median or EWE gold standards, the audio modality emerged as the most significant, while text and video contributed similarly.

An examination of the weights learned by the GRU during fusion showed that when using the mean of annotations as the gold standard, the model assigned equal importance to all modalities. In contrast, with the median and EWE gold standards, the audio modality became the most influential, while text and video modalities contributed similarly to the overall predictions.

## VII. CONCLUSION AND LIMITATIONS

This paper presents the THERADIA WoZ corpus, a multimodal dataset supporting affective computing research in healthcare. Grounded in affective science and appraisal theories, it includes 39.5 hours of multimodal recordings from 52 healthy older adults and 9 participants with MCI, who performed WoZ-assisted cognitive training. To elicit affective responses, inductions techniques were used with 32 healthy

participants. The data were fully transcribed and partially annotated on four appraisal dimensions – novelty, intrinsic pleasantness, goal conduciveness, and coping – as well as arousal, and 23 achievement-related affect labels. Dimensions are available in continuous and summarised forms, highlighting ten core affective labels particularly relevant for AI-assisted cognitive training applications.

Baseline experiments explored affect modeling across text, audio, and video modalities, including multimodal fusion through efficient feature extraction and contextual deep learning methods. Results highlighted important dependencies between modalities and affect representations, providing valuable insights for affect recognition in healthcare. Bridging affective computing and psychology, this multidisciplinary effort advances our understanding of affect and its modeling, benefiting research and practical applications.

However, several limitations should be acknowledged. The fixed annotation order—motivated by the cumulative nature of the Component Process Model—and the uncontrolled at-home annotation settings may have introduced systematic biases, affecting annotation reliability. Moderate inter-annotator agreement further reflects the complexity and subjectivity of annotating subtle affective states in ecological settings, potentially limiting generalizability. Additionally, the decision to avoid crowdsourcing for ethical and training reasons constrained scalability and demographic diversity. Statistical analyses revealed modest yet significant correlations between affective labels and appraisal dimensions, underscoring the limitations of simple linear models in capturing complex affective phenomena. Future research should explore randomized annotation orders, more controlled annotation environments, and privacy-preserving methods to scale annotation ethically. Additionally, adopting non-linear, context-sensitive approaches or qualitative methods may provide deeper insights into these relationships

## VIII. ACKNOWLEDGMENTS

This research has received funding from the Banque Publique d'Investissement (BPI) under grant agreement THERADIA, the Association Nationale de la Recherche et de la Technologie (ANRT), under grant agreement No. 2019/0729, and has been partially supported by MIAI@Grenoble-Alpes, (ANR-19-P3IA-0003).

## REFERENCES

[1] K.-H. Yu, A. L. Beam, and I. S. Kohane, "Artificial intelligence in healthcare," *Nature biomedical engineering*, vol. 2, no. 10, pp. 719-731, 2018.

[2] S. G. Little, J. Swangler, and A. Akin-Little, "Defining Social Skills," in *Handbook of Social Behavior and Skills in Children*, J. L. Matson, Ed. Cham: Springer International Publishing, 2017, pp. 9-17.

[3] H. De Jaegher, E. Di Paolo, and S. Gallagher, "Can social interaction constitute social cognition?" *Trends in Cognitive Sciences*, vol. 14, no. 10, pp. 441-447, Oct. 2010.

[4] L. Pepa, L. Spalazzi, M. Capecchi *et al.*, "Automatic emotion recognition in clinical scenario: A systematic review of methods," *IEEE Transactions on Affective Computing*, vol. 14, no. 2, pp. 1675-1695, 2023.

[5] F. Tarpin-Bernard, J. Fruitet, J.-P. Vigne *et al.*, "THERADIA: Digital Therapies Augmented by Artificial Intelligence," in *Advances in Neuroergonomics and Cognitive Engineering*, H. Ayaz, U. Asgher, and L. Paletta, Eds. Springer International Publishing, 2021, vol. 259, pp. 478-485, series Title: Lecture Notes in Networks and Systems.
[6] N. T. Hill, L. Mowszowski, S. L. Naismith et al., "Computerized cognitive training in older adults with mild cognitive impairment or dementia: a systematic review and meta-analysis," *American Journal of Psychiatry*, vol. 174, no. 4, pp. 329-340, 2017.

[7] H. M. Gavelin, M. E. Domellöf, I. Leung et al., "Computerized cognitive training in Parkinson's disease: A systematic review and meta-analysis," *Ageing Research Reviews*, vol. 80, p. 101671, 2022.

[8] A. Lampit, H. Hallock, and M. Valenzuela, "Computerized cognitive training in cognitively healthy older adults: a systematic review and meta-analysis of effect modifiers," *PLoS medicine*, vol. 11, no. 11, p. e1001756, 2014.

[9] D. Sander, *Models of emotion: The affective neuroscience approach*. Cambridge University Press, 2013, pp. 4–54.

[10] D. Sander, D. Grandjean, and K. R. Scherer, "A systems approach to appraisal mechanisms in emotion," *Neural networks*, vol. 18, no. 4, pp. 317–352, 2005.

[11] A. Moors, "On the causal role of appraisal in emotion," *Emotion Review*, vol. 5, no. 2, pp. 132–140, 2013.

[12] M. Siemer, I. Mauss, and J. J. Gross, "Same situation–different emotions: how appraisals shape our emotions," *Emotion*, vol. 7, no. 3, p. 592, 2007.

[13] K. R. Scherer, H. Ellgring, A. Dieckmann et al., "Dynamic facial expression of emotion and observer inference," *Frontiers in psychology*, vol. 10, p. 508, 2019.

[14] J. Russel, "A circumplex model of emotions," *Journal of Personality and Social Psychology*, vol. 39, pp. 1161–1178, 1980.

[15] H. Fournier and O. Koenig, "Combined effects of intrinsic and goal rel-
evances on attention and action tendency during the emotional episode."  
*Emotion*, vol. 23, no. 2, pp. 425–436, 2023.

[16] J. R. Fontaine, K. R. Scherer, E. B. Roesch et al., "The world of emotions is not two-dimensional," *Psychological science*, vol. 18, no. 12, pp. 1050–1057, 2007.

[17] J. L. Tracy and D. Randles, "Four models of basic emotions: A review of Ekman and Cordaro, Izard, Levenson, and Panksepp and Watt," *Emotion review*, vol. 3, no. 4, pp. 397–405, 2011.

[18] D. Grandjean, D. Sander, and K. R. Scherer, "Conscious emotional experience emerges as a function of multilevel, appraisal-driven response synchronization," *Consciousness and cognition*, vol. 17, no. 2, pp. 484–495, 2008.

[19] K. R. Scherer, M. Mortillaro, I. Rotondi et al., "Appraisal-driven facial actions as building blocks for emotion inference." *Journal of personality and social psychology*, vol. 114, no. 3, p. 358, 2018.

[20] D. M. Low, K. H. Bentley, and S. S. Ghosh, "Automated assessment of psychiatric disorders using speech: A systematic review," *Laryngoscope* investigative otolaryngology, vol. 5, no. 1, pp. 96–116, 2020.

[21] J. Kossaifi, R. Walecki, Y. Panagakis et al., "SEWA DB: A Rich Database for Audio-Visual Emotion and Sentiment Research in the Wild," *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 43, no. 3, pp. 1022–1040, Mar. 2021.

[22] C. Palmero, M. deVelasco, M. A. Hmani et al., "Exploring emotion expression recognition in older adults interacting with a virtual coach," *arXiv preprint arXiv:2311.05567*, 2023.

[23] P. Washington, H. Kalantarian, J. Kent et al., "Training an emotion detection classifier using frames from a mobile therapeutic game for children with developmental disorders," *JMIR pediatrics and parenting*, vol. 5, no. 2, p. e26760, 2022.

[24] I. Zsoldos, E. Trân, H. Fournier et al., "The value of a virtual assistant to improve engagement in computerized cognitive training at home: An exploratory study," *JMIR Rehabilitation and Assistive Technologies*, vol. 11, p. e48129, 2024.

[25] C. E. Ford and S. A. Thompson, "Interactional units in conversation: Syntactic, intonational, and pragmatic resources for the management of turns," *Studies in interactional sociolinguistics*, vol. 13, pp. 134–184, 1996.

[26] I. Eshkol-Taravella, O. Baude, D. Maurel et al., "Un grand corpus oral «disponible»: le corpus d'Orléans 1 1968-2012," *Revue TAL*, vol. 53, no. 2, pp. 17–46, 2011.

[27] H. Brugman, A. Russel, and X. Nijmegen, "Annotating multi-media/multi-modal resources with elan." in *LREC*, 2004, pp. 2065–2068.

[28] R. Pekrun, "The control-value theory of achievement emotions: Assumptions, corollaries, and implications for educational research and practice," *Educational psychology review*, vol. 18, no. 4, pp. 315–341, 2006.

[29] —, "Achievement emotions," *Emotions in late modernity*, vol. 142, 2022.

[30] P. J. Silvia, "Confusion and interest: The role of knowledge emotions in aesthetic experience." *Psychology of Aesthetics, Creativity, and the Arts*, vol. 4, no. 2, p. 75, 2010.

[31] K. R. Scherer, A. Dieckmann, M. Unfried et al., "Investigating appraisal-driven facial expression and inference in emotion communication." *Emotion*, vol. 21, no. 1, p. 73, 2021.

[32] K. R. Scherer, M. Mortillaro, and M. Mehu, "Understanding the mech-
anisms underlying the production of facial expression of emotion: A
componential perspective," *Emotion Review*, vol. 5, no. 1, pp. 47–53,
2013.

[33] J. Cohen, “A power primer.” *Psychological Bulletin*, pp. 155–159, 1992.

[34] K. R. Scherer, H. Ellgring, A. Dieckmann, M. Unfried, and M. Mortillaro, "Dynamic facial expression of emotion and observer inference," *Frontiers in psychology*, vol. 10, p. 508, 2019.

[35] S. Alisamir and F. Ringeval, "On the Evolution of Speech Representations for Affective Computing: A brief history and critical overview," *IEEE Signal Processing Magazine*, vol. 38, no. 6, pp. 12–21, November 2021.

[36] S. Davis and P. Mermelstein, "Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences," *IEEE transactions on acoustics, speech, and signal processing*, vol. 28, no. 4, pp. 357–366, 1980.

[37] G. Salton and C. Buckley, "Term-weighting approaches in automatic text retrieval," *Information processing & management*, vol. 24, no. 5, pp. 513–523, 1988.

[38] P. Ekman and W. V. Friesen, "Manual of the facial action coding system (facs)," *Trans. ed. Vol. Consulting Psychologists Press, Palo Alto*, vol. 3, 1978.

[39] M. Ravanelli, T. Parcollet, P. Plantinga et al., "SpeechBrain: A general-purpose speech toolkit," 2021, arXiv:2106.04624.

[40] T. Baltrušaitis, P. Robinson, and L.-P. Morency, "Openface: an open source facial behavior analysis toolkit," in *IEEE Winter Conference on Applications of Computer Vision (WACV)*, 2016, pp. 1–10.

[41] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, "wav2vec 2.0: A framework for self-supervised learning of speech representations," *Advances in Neural Information Processing Systems*, vol. 33, pp. 12449–12460, 2020.

[42] J. Devlin, M.-W. Chang, K. Lee et al., “BERT: Pre-training of deep bidirectional transformers for language understanding,” in *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*. Association for Computational Linguistics, 2019, pp. 4171–4186.

[43] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., "Learning transferable visual models from natural language supervision," in *International conference on machine learning*. PMLR, 2021, pp. 8748–8763.

[44] A. Conneau, A. Baevski, R. Collobert, A. Mohamed, and M. Auli, "Un-supervised cross-lingual representation learning for speech recognition," in *Proc. Interspeech 2021*, Brno, Czech Republic, 2021, pp. 2426–2430.

[45] C. Sun, L. Huang, and X. Qiu, "Utilizing BERT for aspect-based sentiment analysis via constructing auxiliary sentence," in *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, vol. 1, Minneapolis, Minnesota, USA, Jun. 2019, pp. 380–385. [Online]. Available: https://aclanthology.org/N19-1035

[46] K. Yang, D. Lee, T. Whang et al., "Emotionx-ku: Bert-max based contextual emotion classifier," *arXiv preprint arXiv:1906.11565*, 2019.

[47] F. Weninger, F. Ringeval, E. Marchi et al., "Discriminatively Trained Recurrent Neural Networks for Continuous Dimensional Emotion Recognition from Audio," in *Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI 2016)*. New York City (NY), USA: IJCAI/AAAI, July 2016, pp. 2196–2202.

[48] A. Paszke, S. Gross, F. Massa et al., “Pytorch: An imperative style, high-performance deep learning library,” in *Proceedings of the thirty-third Conference on Neural Information Processing Systems (NIPS)*. Vancouver, BC, Canada: Neural Information Processing Systems Foundation, 2019, pp. 8026–8037.

[49] L. Stappen, L. Schumann, B. Sertolli et al., "Muse-toolbox: The multimodal sentiment analysis continuous annotation fusion and discrete class transformation toolbox," in *Proceedings of the 2nd Workshop on multimodal sentiment analysis challenge*, 2021, pp. 75–82.