BAH Dataset for Ambivalence/Hesitancy Recognition in Videos for  Digital  Behavioural Change
Authors: Manuela González-González
Soufiane Belharbi
Muhammad Osama Zeeshan
Masoumeh Sharafi
Muhammad Haseeb Aslam
Marco Pedersoli
Alessandro Lameiras Koerich
Simon L Bacon
Eric Granger
Venue: ICLR 2026 Poster (2026)
Keywords: Ambivalence
hesitancy
affective computing
emotion recognition in videos
multimodal
eHealth
behavioral change
Primary area: datasets and benchmarks
TL;DR: We introduce new dataset for Ambivalence/Hesitancy recognition in videos with 300 participants and 1,274 videos. Data and code are publically available.

Abstract: Ambivalence and hesitancy (A/H), closely related constructs, are the primary reasons why individuals delay, avoid, or abandon health behaviour changes. They are subtle and conflicting emotions that sets a person in a state between positive and negative orientations, or between acceptance and refusal to do something. They manifest as a discord in affect between multiple modalities or within a modality, such as facial and vocal expressions, and body language. 
Although experts can be trained to recognize A/H as done for in-person interactions, integrating them into digital health interventions is costly and less effective. Automatic A/H recognition is therefore critical for the personalization and cost-effectiveness of digital behaviour change interventions. However, no datasets currently exist for the design of machine learning models to recognize A/H. 
This paper introduces the Behavioural Ambivalence/Hesitancy (BAH) dataset collected for multimodal recognition of A/H in videos. It contains 1,427 videos with a total duration of 10.60 hours, captured from 300 participants across Canada, answering predefined questions to elicit A/H.  
It is intended to mirror real-world digital behaviour change interventions delivered online. BAH is annotated by three experts to provide timestamps that indicate where A/H occurs, and frame- and video-level annotations with A/H cues. Video transcripts,  
cropped and aligned faces, and participant metadata are also provided. Since A and H manifest similarly in practice, we provide a binary annotation indicating the presence or absence of A/H.
Additionally, this paper includes benchmarking results using baseline models on BAH for frame- and video-level recognition, zero-shot prediction, and personalization with source-free domain adaptation methods. The limited performance highlights the need for adapted multimodal and spatio-temporal models for A/H recognition. Results obtained with specialized fusion methods are shown to assess the presence of conflicts between modalities, additionally temporal modelling for within-modality conflicts are essential for more discriminant A/H recognition.
The data, code, and pretrained weights are publicly available: https://github.com/LIVIAETS/bah-dataset.

Published as a conference paper at ICLR 2026
BAH DATASET FOR AMBIVALENCE/HESSTANCY RECOGNITION IN VIDEOS FOR DIGITAL BEHAVIOURAL CHANGE
Manuela Gonzälez-Gonzzller' Soufiane Belharbi', Muhammad Osama Zeeshan', Masoumeh Sharafi' Muhammad Haseeb Aslam' Marco Pedersoli'
Alessandro Lameiras Koerich², Simon L Bacon 3.4 & Eric Granger'
LIVIA, Dept. of Systems Engineering, ETS Montreal, Canada
LIVIA, Dept. of Software and IT Engineering, ETS Montreal, Canada
$ Dept. of Health, Kinesiology, & Applied Physiology, Concordia University, Montreal, Canada
MMontreal Behavioural Medicine Centre, CIUSSS
Canada
[soufianc.. beihhrr., marco. padersol,, alessandro. koorich, grsc.grangers granger)eet.ntl..om
Imhaamad-osamm: zeeshan. masoumeh whwrrai...a............a mchammad htw.ammdddaasssaaaaacaaa....e Bens. .ttmmtl.ca
manuela, gwn.alez Emall concordia, simon. www..aaaoooooo...........
ABSTRACT
Ambivalence and hesitancy (A/H), closely related constructs, are the primary rea- sons why individuals delay, avoid, or abandon health behaviour changes. They are subtle and conflicting emotions that sets person in a state between positive and negative orientations, or between acceptance and refusal to do something. They manifest discord in affect between multiple modalities ity, such as facial and vocal expressions, and body language. Although experts be trained recognize A/H done for in-person interactions, integrating them into digital health interventions is costly and less effective. Automatic A/H recognition therefore critical for the personalization and cost-effectiveness digital behaviour change interventions. However, no datasets currently exist for the design of machine learning models to recognize A/H. This paper introduces the Behavioural Ambivalence/Hesiianyy (BAH) dataset collected for multimodal recognition of A/H videos. It contains 1,427 videos with total duration 10.60 hours, captured from 300 participants across Canada, answering predefined questions to elicit A/H. It is intended to mirror real-world digital behaviour change interventions delivered online. BAH is annotated by three experts provide times- tamps that indicate where A/H occurs, and frame- and video-level annotations with A/H cues. Video transcripts, cropped and aligned faces, and participant metadata are also provided. Since A and H manifest similarly in practice, we provide binary annotation indicating the presence absence of A/H. Addition- ally, this paper includes benchmarking results using baseline models on BAH for frame- and video-level recognition, zero-shot prediction, and personalization with source-free free domain adaptation methods. The limited performance highlights the need for adapted multimodal and spatio-temporal models for A/H recognition. Re- sults obtained with specialized fusion methods are shown to assess the presence of conflicts between modalities, additionally temporal modelling for within within-modalle modality confficts essential for more discriminant A/H recognition. The data, code, and pretrained weights are publicly available: github..om/LLVVAAESSSee
within modal-
INTRODUCTION
Emotion recognition plays a growing role in range of health-related domains (Siddiqi et al. 2024).
including disease prevention (Jin 2024), diagnosis (Jiang al. 2024 Maki al. 2013). treat-

---

Published as a conference paper at ICLR 2026
Body Ianguge: "changing postrc""
Body Iogaage: "locking away"
Figure I: Examples of body language cues used by annotators to identify the occurrence of A/H: "looking away," and "changing posture."
2021 Suraj et al. 2022], and digital health 2022 by supporting adaptive and responsive interventions (Liu et al. [002bb] Sinha et al. 2020). Emotion recognition technologies can support behaviour change interventions (Guo et al. 2024) by identifying affective states relevant to motiva- tion, adherence, and engagement. Health-related behaviour change focuses on strategies to support individuals in adopting and maintaining healthy behaviours to prevent manage diseases, reduce early mortality, and improve mental health and well-being (Davidson & Scholz, (020). Achieving and maintaining long-term behaviour change a complex process. It often includes overcoming am- bivalence and hesitancy (A/H) (McDonald et al., 2002; Michie et al. 201 3a b Voisard et al.) 2024), closely related constructs, which are the primary reasons for individuals to delay, avoid, or abandon health behaviour changes (Conner & Armitage 2008 Conner & Sparks 2002 Manuel & Moyers
ment monitoring (Dhuheir et al. 2021, Pepa et al.
promotion (Arabian et al. |023] Subramanian et al.
2016 Miller & Rose 2015 Van Gent et al. 2024, Williams| 2024).. A/H a subtle and conflicting emotion manifested by a discord in affect between multiple modalities or within modality, such as facial and vocal expressions, and body language. This conflict sets a person in a state between between acceptance and refusal to do something, which of-
positive and negative orientations, ten constitutes a barrier to initiating behaviour change and a trigger for discontinuing interventions or change efforts. Healthcare providers (e.g., clinicians, therapists) often identify A/H through combination of speech and nen-verbal cues (e.g., facial expressions and tone) (Heisel & Mongrain 2004 Labbé al., 2022; Miller & Rose 2015) during in-person interactions. However, integrating them into digital interventions and e-health is costly and less effective. Therefore, designing robust automated methods for A/H recognition can provide cost-effective alternative that can adapt to individual users and operate seamlessly in real-time and shoorrce-liiited environments.
Recent research on machine learning (ML) in emotion recognition focuses mainly on seven basic discrete emotions in images or videos, e.g, 'Happy', 'Sad', and 'Surprised' (Belharbi et al. 2024a Liu et al., 20244) Xue et al., 2022). Other models the literature predict ordinal levels, includ- ing levels linked to health states, e.g., pain, depression, and stress estimation (Aslam et al. 2024
Chaptoukaev et al. 2023 Zeeshan et al., 2024: Nasimzada et al. 2024). Other affects are continu-
ous predictions such as valence-arousa (Dong et al.) 2024 Praveen & Alam 2024a Praveen et al.) 2023 202I]. However, real-world scenarios present more complex cases of emotions. Recently there has been an increased interest in designing robust affect models for compound emotions, case where a mixture of basic emotions is manifested (Kollias) 2023 Richet et al., 2024). In partic- ular, compound emotions commonly occur in daily interactions. Recognizing affective states linked to compound emotions and health states are however, more difficult to discern as they are subtle, ambiguous, and resemble basic emotions. A/H recognition is related to such tasks where intention and attitudes are conflicting or in an intermediate (in-between) state, between willingness and re- sistance (MacDonald] 2015). or positive and negative affect (Armitage & Conner (000). This can manifest in how individuals express themselves and can be recognized (Hayashi et al. (023) in their facial expression, tone, verbal expressions, and body language (Figure D). As result, A/H inherently multimodal, arising from subtle interactions among multiple cues both across modalities and within each modality. Unfortunately, such discord extremely difficult to detect a task that requires human training. This is a tedious and expensive procedure, leading ineffective and less scalable digital interventions under limited resources. Integrating automated and reliable tools for A/H recognition can have major impact on improving digital health interventions. Although A/H is a common topic in behavioural science (Conner& & Armitage 22008] Hohmaan et al. 2016) [anuel & Moyers 2016), it remains unexplored in the MI. community, and such, in the design of eHealth
2

---

Published as a conference paper at ICLR 2026
components. A possible reason is the lack of the necessary and specialized data for training and evaluating ML models.
To address this limitation, this paper introduces first Behavioural Ambivalennc//esitancy (BAH) dataset collected for subject-based multimodal recognition of A/H in videos. Through a collabo- ration between behavioural science and MI teams, have collected large video dataset, BAH, from 9 provinces in Canada. A data capture protocol was set in place to recruit diverse participants, including the development of web platform for video capturing, dedicated storage server, and a specific annotation protocol. Our behavioural science team designed seven questions to elicit re- sponses regarding behaviours and identify possible instances where participants displayed A/H. On our web-platform, participants were presented with these questions and asked to record them- selves while answering. using the camera and microphone on their device. Participants were guided on the platform by an avatar throughout the entire data capture session. Our dataset was developed with weal-world digital health applications in mind, particularly for use in personalized behaviour change interventions where users interact with an avatar that prompts them with predefined ques- tions. To mirror this setting, participants in our study responded to structured but genuine questions about behaviours they engage in avoid, based on their actual experiences. This setup encourages authentic, spontaneous expression of ambivalence and hesitancy (A/H), while still maintaining con- sistency across participants. The BAH dataset was intentionally designed to maximize ecological validity. Participants had control over their environment and delivery, which contributed to natural- istic responses that reflect how people express A/H in real life.
The BAH dataset is composed of videos from 300 participants. This amounts to a total of 1,427 videos (~ 10.60 hours) where 778 videos contain A/H (~~ 1.79 hours). It has 916,618 total frames, where 156,255 contain A/H. Three of our behavioural experts annotated the data at the video- and frame-level to assess when A/H occurs. This also provides timestamps indicating when A/H starts and ends, allowing temporal localization. Because ambivalence and hesitancy manifest sim- ilarly in practice, the annotation is framed under a binary form (A/H vs. non-A/H). The video cues used by the annotators are also reported, such as facial expressions, body language, audio and language, in addition to highlighting where there is inconsistency between the modalities. The BAH dataset is made public, and it is provided with the raw videos with audio, cropped and aligned faces, detailed annotation/cues at the video- and frame-level, A/H timestamps, audio tran- acrip//imesttaps/languuee and participant metadata such as age, ethnicity, etc. Part of BAH has already been used in a public challenge at the 2025 (Hallmen et al.] 2025) Kollias et al., [2025] Savchenko 2025) and 2026 'Workshop and Competition on Affective & Behavior Analysis in-the- wild (ABAW) (8th and 10th ABAW).
Our main contributions are summarized as follows. (1) A novel video dataset named BAH is proposed for automated participant-based and multimodal recognition of A/H in videos that can be used design ML models for digital health intervention systems. To mirror real-world behaviour change interventions, 300 participants in our study recorded themselves while responding to struc- tured but genuine questions about behaviours that elicit A/H using our online platform. Behavioural experts labelled the data at the video- and frame-levels to indicate the presence/absence of A/H (since they are manifested similarly). BAH can be used to develop and evaluate standard and person- alized MI. models for classification and localization tasks, and build insights about A/H for digital behaviour change interventions. (2) Preliminary benchmarking results using baseline models on BAH for frame- and video-based emotion recognition. Results allowed exploring the impact of key factors, including the impact of using temporal context, multimodal information, and feature fusion. They also showed the need for specialized modalities fusion, and temporal modelling for better performance in the new task of A/H recognition. Baseline results are also shown for other tasks zero-shot prediction and personalization (source-free domain adaptation) through subject-based domain adaptation. Our code and dataset are publicly available.
2
THE BAH DATASET
2.1 DATASET COLLECTION AND ANNOTATION
Capture. The BAH dataset contains Q&A videos. It is constructed by collecting samples from par- ticipants over the age of 18 across Canada. The data collection and annotation process is presented in Figure 2 To proceed with the data collection, we developed an "Automatic Expression Recogni-
3

---

Published as a conference paper at ICLR 2026
Our online platform for data collection
Video annotation for A/H at multiple levels: video- frame-lesel, cues
acrsttdy.ca
Recording
Data transfer by Administralor
Secured access
Our local secured storage server for videos
Participant
Task: answer the designod questions while recording themselies via webcam
Our behavioural expeat annolator
BAH dataset collection and annotation procedure. First, participants access web plat- form. They undergo an initial rest/calibration to ensure the quality of the data. An avatar guides them throughout the entire process. Seven questions are presented to each participant. They recorded while answering them. Once the data is captured, is transferred by the Administrator to local server. It is then annotated several levels by experts to determine when A/H occurs.
Figure 2:
tion" (AER) web-based platform (www.aerstudy.ca where participants could record their responses to specific questions using their own computers or devices with a camera and a microphone. Users received secure credentials to access the data collection platform, or they created their own account. Participants first complete a brief survey to provide demographic information and indicate consent preferences (e.g., inclusion in secure datasets, challenges, or publications). They were then redi- rected to the AER platform, where they tested their camera and microphone and chose an avatar to interact with during data capture. The avatar guides them through seven questions. The session took approximately 30 minutes. Participants were recruited and compensated through the Prolific company (www.prolifcc.com) which also ensures population diversity and allows submission pro- cessing.
Participants answered seven questions designed by our behavioural team (Table [ each one in- tended to elicit neutral, positive, negative, ambivalent, willing, resistant, and hesitant answers. Once the question was presented, the recording of the participant's 's response started. Skipping questions was allowed. At the end of each question, the participant has the option to rate their emotional response using a 5-point Likert-like scale. This self-rating was only employed for our analysis and did not serve as an annotation. The order of the questions was randomized. In addition, participants were not aware of what each question was expected to elicit as emotion. During this capture proce- dure, several pieces of information were gathered, including contact information of the participant, their demographics, consent, video recordings, survey responses, and software usage data (such as the time spent on each question). The software used to interact with the participants and capture their data was managed by the MBMC and the Concordia research team. The participants' data was systematically downloaded and transferred to a local secure server storage (located at ETS Montreal) by the team for annotation and further analysis.
The study obtained human ethics approval from both Concordia University (www.concordia.cal (certification number: 30019002) and LET((www.esmmll.cm (certification number: H202312033 and the study followed all standard ethical practices. The dataset was collected by researchers from Concordia University and the Montreal Behavioural Medicine Center (MBMC) in collaboration with the Laboratoire d' d'imagerie, de vision et d' d'intelligence artificielle (LIVIA) at ETS Montreal. The dataset was collected between September 2024 and April 2025 in batches. This allowed to adjust the targeted population (regarding participants' sex and Canadian province of residency) to ensure the dataset diversity.
We note that while the BAH dataset is sourced exclusively from Canadian participants, its sample collection was designed to be representative in terms of province and sex, two key demographic variables relevant to health behaviour research in Canada. Additionally, the dataset includes in- dividuals from a wide range of backgrounds, including different ethnicities and national origins, which reflects the cultural diversity of the Canadian population. This diversity, often lacking in many existing datasets, enhances the relevance and applicability of our findings within the Canadian context. Despite the geographic limitation, we consider this dataset meaningful starting point for
4

---

Published as a conference paper at ICLR 2026
understanding ambivalence and hesitancy in health-related behaviour. We plan to expand other countries and cultural settings to support broader generalizability.
Annotation. Three annotators were trained in expression recognition, specialized in identifying A/H, and in the annotation process of audio-visual data. A two-stage process was used: first, a global-level annotation determined the presence of A/H in each video; then, a frame-level annotation identified the precise segments where A/H occurred, specifying the start and end times (i.e., onset and offset) of each instance. Annotators also provided certainty ratings, and for some segments, indicated the cues that supported their judgment. To identify A/H, annotators tracked expressions across well-established modalities (facial expressions. body language, audio, and language) and flagged cases where inconsistencies between modalities were observed. Each A/H segment has a timestamp start/end in addition to its own cues and modalities determined by an annotator, allowing for better precision. Multiple A/H segments could be present within video. We do not include an "apex" nor a continuous annotation, as ambivalence and hesitancy do not reliably exhibit a peak moment of maximum intensity. Instead, they tend to manifest as sustained or fluctuating states, making the concept of an apex incompatible with their typical temporal structure. The videos were annotated following a codebook created specifically for the study. Videos were annotated using the software (Figure 2
ELAN 8 aurchiee.mi..m//lleekee
The annotation process followed structured training protocol supported by a detailed training man- ual. Annotators first received a conceptual introduction to A/H, followed by hands-on training in using the ELAN annotation software. Practical application was conducted using a standardized set of videos from the dataset. This phase also introduced annotators to the codebook, emphasizing the cue list, with examples spanning facial, vocal, verbal, and bodily expressions. Annotators received feedback, and additional sessions were provided when further alignment was needed. Only after this training phase, and a final assessment, did annotators proceed to independent annotation. To ensure large labeled data, BAH was divided among the three annotators, where each video is labeled by one annotator. However, we analyzed over 10 random videos with a multi-annotann sess inter-annotator agreement. We considered Fleiss's Kappa measure (Fleiss) 1971) for more than 2-anmotator cases. At the global level, the score measure was 0.65 (substantial agreement), while at the frame level, we had 0.41 (moderate agreement). When considering only videos where all three annotators agreed they had A/H, the score was 0.50.
setup to as-
To promote consistency, annotators
instructed to flag cases of uncertainty complexity. These cases were discussed collaboratively, often through co-annotation. A consistent lead annotator fa- cilitated resolution efforts, ensuring that decisions reflected shared interpretation. In parallel, comprehensive annotation protocol guided how videos were managed, accessed, and annotated. An- notators followed standardized procedures: (1) watch the video without taking notes to understand the participant and context; (2) re-watch the video to identify A/H segments and record start and end times; (3) reassess and refine selected segments; (4) identify and assign cues using the codebook; and (5) if needed, watch the video without audio or visual elements to isolate specific signals. Anno-
Question no. Response
Prompt
Neutral
Tell us about an activity you commonly do after waking up.
2
Positive
Talk about an activity that brings you joy, for example, a hobby.
Tell why.
3
Negative
Talk about an activity you dislike doing. for example, chore something you find boring annoying. Tell us why.
Ambivalent
Tell about something you enjoy doing but wish you stopped do- ing (like guilty pleasure) something you don't do but wish you did.
5
Willing
Tell us about activity you are almost always willing do, for example, with friends, at work, at home.
6
Resistant
Tell about something people around you do, but that you would not be willing to do. for example, with friends, work, home.
7
lesitant
Tell about something you could have done already but haven't done yet, for example, something you are procrastinating or haven't made up your mind about.
Table 1: The 7 questions (prompts) designed by our behavioural science experts to capture videos for the BAH dataset. To avoid influencing the participant's 's answers, they are only shown prompts without indicating the expected emotion/response.
5

---

Published as a conference paper at ICLR 2026
tators were also encouraged to consult other videos from the same participant to establish expressive baselines in ambiguous cases.
The presence of A/H is assigned single label (I), while its absence is assigned the label O. Each video has a global- and frame-level label which can be used to train and evaluate ML models. The provided cues can also be used for the interpretability aspect, as well as to build insights into how people express A/H. The dataset is structured participant- wise, which can also be useful for personalization training scenarios. The BAH dataset that is being made public contains only videos of participants who consented for their data to be made public.
2.2 DATASET VARIABILITY
The BAH dataset is designed to approximate the demographic distribution of sex and provincial representation in Canada. It is composed of 300 participants across Canada from nine provinces, where 25.7% of participants are from British Columbia, followed by Alberta with 19.7% and Ontario with 17.3% D All participants agreed to be part of this dataset. However, 61 participants (20.3% P did not consent to be in publications, while only seven participants (2.3%) did not consent to be part of challenges. The majority of recorded videos are in the English language, and very few are in the French language. Each participant can record up to seven videos, and 113 participants have recorded the full seven videos. We obtained an average of 4.75 videos/participant where each participant has an average of ~ 2.59 videos with A/H which is equivalent to ~ 520.85 frames of A/H (or ~ 21.49 seconds of A/H). The dataset amounts a total of 1,427 videos (~ 10.60 hours) where 778 videos contain A/H (~ 1.79 hours). This amounts to 916,618 total frames, where 156.255 contain A/H. Since captured videos represent answers to questions, they are relatively short. BAH dataset has an average video duration of 26.76 + 16.47 (seconds) with minimum and maximum duration of 3 and 96 seconds.
An important characteristic of BAH is the duration of A/H segments in videos. BAH dataset counts total of 443 videos with multiple A/H segments and 332 videos with only one A/H segment. In total, there are 1,504 A/H segments. In particular, the duration of segments varies, but it is brief with an average of 4.29 + 2.45 seconds, which is equivalent to 103.89 + 58.70 frames. The minimum and maximum A/H segment is 0.004 seconds (1 frame) and 23.8 seconds (572 frames), respectively.
In terms of participants' age, the dataset covers a large range from 18 to 74 years old. In particular, 37.7% of the participants cover the range 25-34 years, followed by the range of 35-44 years with 24.3%, then the range of 18-24 years with 20.7%. In terms of sex, 52.0% are female, while 47.3 are male. As for ethnicity variation, 54.0% of the participants identified as "White", followed by "Asian" with 21.0%, and "Mixed" with 10.7%, then "Black" with 9.7%. The majority of the partic- ipants were not students (67.0%), which limits the common issues in recruit bias in other datasets.
The public BAH dataset contains the row videos, detailed A/H annotation at video- and trame-level, timestamps of A/H, cues, script to extract all frames, and per participant demographic information, including age, birth country, Canadian province where the participant lived, ethnicity, ethnicity sim- plified, sex, student status, consent to use recordings in publications. More details about the dataset diversity are provided in the appendix.
2.3 ETHICAL CONSIDERATION DATASET ACCESSIBILITY AND INTENDED USES
The collected data of human participants closely follows standard ethical considerations. The project to collect BAH data was approved by ethical committees from both collaborating universities, Con- cordia University under agreement n 30019002 and ETS under agreement n H20231203 Once recruited, participants had access to the full consent form prior to accessing the data capture plat- form and starting their data capture procedure. They were provided with details of the study, as well as a list of the potential risks and benefits of participating in the study. They were asked to read the consent form thoroughly, and they were provided with clear and simple video that summarized the consent form. Participants were then able to decide the type of access they wanted researchers to have to their audiovisual data, including whether they wanted their images to be used for publica-
'Provinces: 'Manitoha (MB), 'Alberta (AB)', 'Nova Scotia (NS),, 'Newfoundland and Labrador (NI.), "Saskatchewan (SK)', 'New Brunswick (NB),, 'Ontario (ON)', "Quebee (QC)', 'British Columbia (BC)'. "The list of these participants is provided within the shared files of the BAH dataset.

---

Published as a conference paper at ICLR 2026
tions and presentations. In addition, these options were presented again at the end of the data capture procedure, just in case they changed their minds about their participation in the study or the use of their data after they had finished recording their responses. At the end of the study, participants receive, via email, a copy of the consent form that included their choices about data usage and the contact information for the team should they have any further questions. Participants were given numerical codes for anonymity.
Following the guidelines of the funding agency of the Canadian government, the BAH dataset is made public with open credentialed access for research purposes. To access the dataset, users are required fill in request form and sign an End-User License Agreement (EULA) as commonly done to ensure dataset security. Upon access approval, the user will receive a link to download the full dataset, including row videos, detailed annotation, cues, participants' metadata, cropped- and-aligned- faces, and audio transcripts. BAH uses proprietary license for research purposes. The dataset is hosted on secure server at ETS is intended for long-term availability. Our public code github.com/LLIv www.////AATT//aaa.ooooooooo
is under an open-source license (BSD-3-Clause license). The code website will be used as permanent page for the dataset that will reflect any future updates. Despite all our precautions, our dataset may still be misused. We consider thorough review of requests before granting data access.
Part of the BAH dataset has already been used by the public for the Ambivalencc//Hesitnnyy (AH) Recognition Challenge organized in the "8th and 10th Workshop and Competition on Affective & Behavior Analysis in-the-wild (ABAW) in conjunction with CVPR 2025 and 2026" (Hallmen al.] 2025 Kollias et al. 2025
Savchenko 2025). A similar access protocol was implemented.
Our primary goal of building the BAH dataset was make public a first and unique dataset for A/H recognition in videos. Given the content of the dataset, its multimodal aspect, and the provided annotations, it can be used train and evaluate ML models for A/H recognition in videos at frame- and/or video-level with different learning scenarios. Since data is individually collected from par- ticipants, it can also be used for model personalization using domain adaptation. The cues used by annotators can also be used to learn interpretable models, providing further insight into our under- standing of A/H in human behaviours. Such understanding and recognition of A/H can be leveraged in downstream tasks such as behavioural change, interventions and recommendations in clinics or through automated systems such as virtual trainers/assistants.
2.4 EXPERIMENTAL. PROTOCOL
Dataset split. The dataset is divided randomly based participants into 3 sets: train (195 par- ticipants), validation (30 participants) and test (75 participants) sets. We ensured that the 3 splits represent the total data distribution. The train and validation sets amount to 3/4 of the total par- ticipants, while 1/4 goes to the test set. Videos of one participant belong to one and one set only. The details of each set are presented in Table 2 The split files are provided along with the dataset files. They contain the split in terms of videos and frames, ready to use. Note that the dataset is highly imbalanced as depicted in Table especially at the frame level, where only 17.04% contains A/H. This factor should be accounted for during training and evaluation. The dataset can be used for training the video- and/or frame-level. The participant identifiers are provided in the splits, allowing subject-based learning scenarios.
Data subsets
Train
Validation
Test
Total
Number participants
195
30
75
300
Number participants with A/H
144
27
75
246
Number videos
778
124
525
1427
Number videos with A/H
Number frames
501.970
79,538
335.110
916.618
Number frames with A/H
76,515
13,984
65.756
156,255
Total duration (hour)
5.80
0.92
3.87
10.60
Total duration with A/H (hour)
0.87
0.16
0.75
1.79
Table 2: BAH dataset split into train, validation, and test sets.

---

Published as a conference paper at ICLR 2026
Data suhsets
Train (8)
Validation (%)
Test (%)
Total
Participants with A/H
73.85
90.00
100.00
82.00
Videos with A/H
49.49
60.48
60.57
54.51
Frames with A/H
15.24
17.58
19.62
17.04
Duration with A/H
15.09
17.41
19:44
16.88
Imbalance rate of BAH dataset split across train, validation, and test sets: (Total # items with A/H)/(Total # items).
Table 3:
Evaluation metrics. We refer here to the positive class as the class with label 1, indicating the presence of A/H, while the negative class is the class O, indicating the absence of A/H. To account for the imbalance in the BAH dataset, we use adapted standard evaluation metrics: Average Fl (AVGF1) score, which is the unweighted mean of F1 of the positive and negative classes. Average precision score (AP) of the positive class, which accounts for the performance sensitivity to the model's confidence. For score, a threshold list between and I is used with step of 0.001. Evaluation code of all measures is provided along with the public code of this dataset.
3 BASELINE RESULTS
This section provides preliminary results of different baseline models on our BAH dataset. In partic- ular, we provide the performance of models for the supervised frame-level classification task. We consider 2-class classification problem where each frame is annotated, and models predict two outputs: one for the positive class (presence of A/H), and a second for the absence of A/H. Super- vised level classification performance is included in the appendix in addition to other tasks zero-shot prediction, and personalization through unsupervised domain adaptation.
We initially focus the impact of using single multimodal learning for frame-level classification. Then, the performance of different individual modalities are explored, along with their multimodal fusion. In addition, we investigate the impact of temporal modelling and context single frame learning. Next, we present the pre-processing of the three different modalities used: visual (facial), audio (vocal), and text transcripts (textual), and describe the baseline models used in each case.
3.1 PRE-PROCESSING OF MODALITIES
1) Visual. All frames from each video extracted, and for each frame, faces are located using the RetinaFace model (Deng et al. 2019), cropped, and then aligned. The face with the highest score is stored in case multiple faces detected in frame. Faces resized 256 256 and stored as RGB images with a file name that maintains the order of frames. The video frame rate is 24 FPS. 2) Audio. We follow standard procedure process audio data (Praveen Alam 2024b Richet al.,
2024 Zhang et al. 2023b). For audio modality. we first convert videos to single audio channels (mono) with a 16k sampling rate into wav format. The log melspectrograms features are extracted using the VGGish model (Hershey et al. (github. crathu..com/aaritaaa wowprritaayloooorryygggy gggisggg A hop of 1/FPS of the raw video used to extract the spectrograms to synchronize audio with other modalities. 3) Text. The collected data captures the audio of participants. We consider audio transcripts as an extra modality that can help in recognizing A/H since text is significant cue used by annotators. To this end, we transcribe the audio of each video and detect the language using the Whisper model (Rad- ford et al. 2023) (Whisper large-v3 multilingual: provide the timestamp of each transeript. Word-level features are then extracted using the BERT Base Uncased model (Devlin et al. span more than one frame. To synchronize with other modalities, a word-level feature repeated per its timestamp for all the frames that correspond to the word.
We
porettoreeeerrined--aaaee bert/). A word may
Note that both cropped and aligned faces, and video transcripts are shipped with the shared public BAH dataset. Researchers can choose to use them or build their own.
3.2 PRE-TRAINING OF VISUAL BACKBONE
For audio and text modality, features are extracted offline and stored as described above. For vi- sual modality, we explore different architectures, including CNN- and ViT-hased (Dsovitsski al.)
8

---

Published as a conference paper at ICLR 2026
Without context
With context (TCN)
Backbone
AVGF1
AFGF1
APVIT Xue et
2022
0.5051
0.1906
0,5019
0.2069
(8 ResNet
2016
0.5074
0.1940
0.5079
0.1993
ResNet34
[e et
2016
0.5138
0.1952
0.4998
0.1984
ResNer50
(He et
2016
0.4737
0.1942
0.4985
0.1915
ResNet101 [Hc al.
2016
0.4929
0.1967
0,5165
0.2070
(He R5sNet152
al.
2016
0,4889
0.1843
0,5084
0.2058
Table 4: Performance of the visual modality (facial expressions) on the test subset of BAH for frame- level classification. It indicates the impact of architecture and context.
20211. In particular, we explore the ResNet family, including ResNet18, 34, 50, 101. and 152 (He et al. 2016). For the VIT family, we consider recent model designed for basic emotion recog- nition, APVIT (Xue et al., 2022). First, we pre-train each model on basic emotion recognition task, including these emotions: "Anger". "Disgust", "Fear", "Happiness", "Sadness", "Surprise". To this end, we collected a large mixed dataset composed of public common datasets for emotion recognition using images: RAF-DD DD (Li et al. 2017). Af Net (Mollahosseini et al. (2019), and Aff-wild2 (Kollias & Zafeiriou 2019). This amounts to more than 0.54 million training images. Models are trained for basic emotion classification for 60 epochs with batch size of 1,424 sam- ples using 4 parallel NVIDIA A 1100 GPUs with 40 GB of memory. Standard cross-entropy loss and Stochastic Gradient Descent (SGD) are used for training. Once pretrained, each model is further fine-tuned on our BAH train set for A/H recognition. To account for class imbalance, we perform sader-sampling of the negative class over the training set. This is achieved by randomly sampling the negative class samples to be the same as the positive class. The weights of both models are made public. The backbones of each model are used later for feature extraction of the visual modality. We compared variant visual modalities, including cropped faces, full-frame, and head-pose posee. Results show that cropped faces yield better results. Details are included the supplementary materials.
3.3 IMPORTANCE OF CONTEXTUAL LEARNING
In this section, we aim to answer the question of whether context modelling help better A/H recognition. This is particularly interesting since A/H does not occur instantly. but within a context. Text and audio modality already capture context in their features, making using them to answer this question less efficient. However, we can obtain frame features with and without any context. Therefore, we consider visual modality with different backbones to answer our question.
In the case of modelling without context, models simply train on independent frames without consid- ering any context or dependency between them. In- ference done in the same way. In the case of modelling with context, both training and inference leverages temporal dependency between frames. To this end, window of adjacent frames is fed to the model. We then use a temporal convolutional network (TCN) (Bai al. 2018) after the visual backbone to capture relations between frame embed- dings. The window length defines the extent of the context. Table showss the obtained results. Regard- less of the context, we observe a very low perfor- mance over AP, highlighting the difficulty of recog- nizing A/H based on images alone. In particular, AP is below 0.2070. On the other hand, AVGFL 1 reaches 0.5138. As a reference, predicting every frame as the negative class yields an AVGF1 of 0.4459. Overall, using context boost performance of all metrics across all architectures. This is expected as A/H does not usually occur at a single frame but within context. This makes its recognition from single frame challenging. We recall that the average A/H segments span 103 frames (or 4.29 seconds). Future work should account for the temporal context for better performance. However, as we will show in the ablations in the appendix, a very large context could lead to poor performance. Note that large ResNet models seem to yield better performance overall. Unless mentioned otherwise, all next experiments will use 10.
Prediction of A/H
Classification
Feature-level fusion
Textual
Audio features
Visual features
features
Feature extraction
Feature extraction
Feature
extraction
"I' still cosfisee about
when I'm going ofter taulies
Text transcript
Audio
Visual
Figure 3: Multimodal model used to produce baseline performance (Richet et al. 2024).

---

Published as a conference paper at ICLR 2026
3.4 MULTIMODAL BASELINES
Since A/H is multimodal data, we explore the impact of
using different modalities, including visual, audio, and
transcript. using the model shown in Figure 3 (Richet
Modalitics
AVGF1
AF
et al. (024). Results are reported in Table Using
Visual
0,5165
0.2070
the text modality alone yields better performance, AP of 0.2510 and AVGF1 of 0.5497, compared to visual or text modalities over both metrics. Combining a pair of modalities improves the performance to 0.5586 for AVGF1, and 0.2609 for AP, in the case of audio and text. Combining the three modalities slightly reduces perfor- mance in terms of AP, suggesting that more adapted fu- sion techniques needed recognize affect conflicts between/within modalities.
Audio
0.4658
0.2238
Text
0.5497
0,2519
Visual + Audio
0.5205
0.2225
Visual + Text
0.5547
0,2479
Audio Text
0.5586
0.2609
Visual + Audio Text
0,5502
0.2548
Table 5: Frame-level classification per-
formance of multimodal models BAH.
ResNet101 is used for the visual modal- ity.
Table 6 shows the impact of using different fea- ture fusing techniques including simple concatenation
(CAN) (Zhang al., 2023b), co-attention (LFAN) (Zhang et al. 2023b), transformer-based fusion (MT) (Waligora et al. (0224), and cross-attention fusion (JMT) (Waligora et al.) (024)). We observe that the way of leveraging the interaction between the three modalities is a key factor. LFAN and CAN fusion lead over both metrics. Future works should pursue more adapted methods to A/H. Since A/H are usually expressed as a conflict between willingness and resistance, they can be per- ceived through parallel affect conflict between modalities and or within modalities. For instance, a participant could say a sentence to convey a meaning but their facial expression, body behaviour, or tone may carry contradictory emotion. Understanding such subtly and interconnection between different cues in different modalities could play an important role in designing robust methods for A/H recognition in videos.
We believe our new and unique dataset has brought a new, challenging research direction to better understand complex and subtle human emotions that is A/H. Given the multimodal nature of A/H, our BAH dataset provides an essential and valuable toolkit for the research community to design and evaluate their methods. Important key and critical downstream tasks could potentially benefit from these methods, including but not limited to clinical interviews, interventions, behavioural changes, and automated assistants such as online trainers. Our preliminary results suggest that leveraging context, multimodality, and their fusion could lead to better A/H recognition performance.
Method
Fusion Appreach
AVGF1
AP
LFAN IZhan cl
2023b
(cypmr22023)
Co-sitcation
0.5502
0,2548
CAN Zhangg
21255
(023)
Concstenation
0,5526
0.2631
MT
Wargora et al
2P4
(cpp);202 ()
Transformer
0,5137
0,2134
JMT
Waangora
(c)pno22024)
attentin
0.5241
Table 6: Impact of feature fusion methods on test set of BAH at frame-level classification.
4 CONCLUSION
This paper introduces BAH, new multimodal and participant- dataset for A/H recognition in videos. It contains the videos of 300 recruited participants captured across 9 provinces in Canada. Participants recorded themselves using webcam and a microphone through our web web-platform while they answered 7 questions designed to elicit A/H. The dataset amounts to 1,427 videos for a total duration of 10.60 hours, with 1.79 hours of A/H. was annotated by our behavioural science team at the video- and frame-level. Our initial benchmarking study yiclds limited performance, highlight- ing the difficulty of A/H recognition. Results also indicate that leveraging context, multimodality, domain adaptation and adaptive feature fusion promising directions improve the accuracy and robustness of ML models on BAH. Our dataset and code are made public.
The supplementary materials contain recommendations for future work about A/H recognition, lated work, more detailed and relevant statistics about the dataset and its diversity, dataset limita- tions, implementation details, and additional results.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was supported in part by the Fonds de recherche du Québec Santé (FRQS), the Natural Sciences and Engineering Research Council of Canada (NSERC), Canada Foundation for Innovation (CFI), and the Digital Research Alliance of Canada.
REFERENCES
H. Arabian, T. Abdulbaki Alshirbaji. R. Schmid, V. Wagner-Hartl, J. Chase, and K. Moeller. Har- nessing wearable devices for emotional intelligence: Therapeutic applications in digital health. Sensors, 23(19):8092, 2023.
C. Armitage and M. Conner. Attitudinal ambivalence: of three key hypotheses. Personality and Social Psychology Bulletin, 26(11):142114) 2000.
M.H. Aslam, M.O. Zeeshan, S. Belharbi, M. Pedersoli, ALI.. Koerich, S. Bacon, and E. Granger. Distilling privileged multimodal information for expression recognition using optimal transport. In International Conference on Automatic Face and Gesture Recognition (FG), 2024.
S. Bai. J.Z. Kolter, and V. Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. CoRR, abs/ 100s/1803.1171,, 2018.
S. Belharbi, M. Pedersoli, A. L. Koerich, S. Bacon, and E. Granger. Guided interpretable facial expression recognition via spatial action unit cues. In International Conference Automatic Face and Gesture Recognition (FG), 2024a.
S. Belharbi, M. Pedersoli, A.L. Koerich, S. Bacon, and E. Granger. Spatial action unit cues for interpretable deep facial expression recognition. Al and Digital Health Symposium, 2024b.
M. Bishay, P. Palasek, S. Priebe, and Patras. Schinet: Automatic estimation of symptoms of schizophrenia from facial behaviour analysis. IEEE Transactions on Affective Computing, 12(4): 949 961, 2019.
Bonnard, A. Dapogny, F. Dhombres, and Bailly. Privileged attribution constrained deep net- works for facial expression recognition. In ICPR, 2022.
E.H. Bradley, L.A. Curry, and K.J. Devers. Qualitative data analysis for health services research: developing taxonomy, themes, and theory. Health services research, 42(4):1758-177,, 2007.
C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J.N. Chang, Lee, and S.S. Narayanan. lemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42:335-359, 2008
H. Chaptoukaev, V. Strizhkova, M. Panariello, B. Dalpaos, A. Reka, Manera, S. Thummler, E. ISMAILOVA, N. Evans, F. Bremond, M. Todisco, M. A Zuluaga, and L. M. Ferrari. StressID: multimodal dataset for stress identification. In NeurlPS, 2023.
J. Choe, S.J. Oh, S. Chun, S. Lee, Z. Akata, and H. Shim. Evaluation for weakly supervised object localization: Protocol, metrics, and datasets. TPAMI, pp. 1-1, 2022.
Y. Chu, L. Liao, Z. Zhou, C.-W. Ngo, and R. Hong. Towards multimodal emotional support conver- sation systems. CoRR, Abs/2408.036500 2024.
M. Conner and C.J. Armitage. Attitudinal ambivalence. 2008.
M. Conner and P. Sparks. Ambivalence and attitudes. European review of social psychology, 12(1): 37-70, 2002.
K.W. Davidson and Scholz. Understanding and predicting health behaviour change: a contem- porary view through the lenses of meta-reviews. Health psychology review, 14(1):1-5, 2020.
M. De-la Torre, E. Granger, P. VW Radtke, R. Sabourin, and D.O. Gorodnichy. learning from facial trajectories for face recognition in video surveillance. Information fustoaa,ioontssion,, 24: 31-53, 2015.
11

---

Published as a conference paper at ICLR 2026
J. Deng, J. Guo, Y. Zhou, J. Yu, I. Kotsia, and S. Zafeiriou. Retinaface: Stngle-stage dense face localisation the wild. CoRR, 1es/1905500641, 2019.
J. Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: pre-training of deep bidirectional trans- formers for language understanding. In NAACL-HLI, pp. 4171 4186,, 2019.
M. Dhuheir, A. Albaseer, F.. Baccour, A. Erbad, M. Abdallah, and M. Hamdi. Emotion recogni- tion for healthcare surveillance systems using neural networks: A survey. In 202/ International Wireless Communications and Mobile Computing (IWCMC), pp. 681-687, 2021.
L.. Dong, X. Wang, Setlur, V. Govindaraju, and I. Nwogu. Ig3d: Integrating 3d face representa- tions in facial expression inference. CoRR, 1ess22408.16907,, 2024.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, Uszkoreit, and N. Houlsby, An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
Y. Fan, J. Lam, and V. Li. Facial action unit intensity estimation via semantic correspondence learning with dynamic graph convolution. In AAAI, 2020.
J.L. Fleiss. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5): 378, 1971.
R. Guo, H. Guo, L. Wang, M. Chen, D. Yang, and B. Li. Development and application of emotion recognition technology-a systematic literature review. BMC psychology, 12(1):95, 2024.
X. Guo, B. Zhu, L.F. Polanía, C. Boncelet, and K.E. Barner. Group-level emotion recognition using hybrid deep models based faces, scenes, skeletons and visual attentions. In Proceedings of the 20th ACM international conference on multimodal interaction, pp. 635-639, 2018.
J.A. Hall, J.A. Harrigan, and R. Rosenthal. Nonverbal behavior in clinician-patien interaction. Applied and preventive psychology, 4(1):21-37, 1995.
T. Hallmen, R.-N. Kampa, F. Deuser, N. Oswald, and E. André. Semantic matters: Multimodal features for affective analysis. In ABAW workshop at CVPR, 2025.
J. Han, L. Xie, Liu, and X. Li. Personalized broad learning system for facial expression. Multi- media Tools and Applications, 2020.
D. Hayashi, S.D.L. Carvalho, P.A.B. Ribeiro, R.C.M. Rodrigues, T.M. São-João, K. Lavoie, Ba- con, and M. E. Comelio. Methods to assess ambivalence towards food and diet: a scoping review. Journal of Human Nutrition and Dietetics, 36(5):2010-203) 2023.
K. He, X. Zhang, S.g Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
Z. He, Z. Li. F. Yang, L. Wang, J. Li, C. Zhou, and J. Pan. Advances in multimodal emotion recognition based on brain-computer interfaces. Brain sciences, 10(10):687, 2020.
M.J. Heisel and M. Mongrain. Facial expressions and ambivalence: Looking for conflict in all the right faces. Journal of Nonverbal Behavior, 28:35-52, 2004.
S. Hershey, S. Chaudhuri, D.P.W. Ellis, J.F. Gemmeke, A. Jansen, R.C. Moore, M. Plakal, D. Platt, R.A. Saurous, B. Seybold, M. Slaney, R.J. Weiss, and K. W. Wilson. Cnn architectures for large- scale audio classification. In ICASSP, 2017.
Z.P. Hohman, W.D. Crano, and E.M. Niedbala. Attitude ambivalence, social norms, and behavioral intentions: Developing effective antitobacco persuasive communications. Psychology of Addictive Behaviors, 30(2):209, 2016.
S. Hornstein, K. Zantvoort, U. Lucken, B. Funk, and K. Hilbert. Personalization strategies in dig- ital mental health interventions: a systematic review and conceptual framework for depressive symptoms. Frontiers in digital health, 5:1 170002, 2023.
G. M. Jacob and B. Stenger. Facial action unit detection with transformers. In CVPR, 2021.
12

---

Published as a conference paper at ICLR 2026
D. Jiang, L. Yan, and F. Mayrand. Emotion expressions and cognitive impairments in the elderly: review of the contactless detection approach. Frontiers in Digital Health, 6:1335289, 2024.
H. Jin. comparative analysis of single and multi colti-moddlity-bbane emotion recognition for disease prevention. In International Conference Artificial Intelligence and Communication (ICAIC), volume 185, pp. 323, 2024.
W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanaraiinhan, F. Viola, T. Green, T. Back, P. Natsev, and M. Suleyman. The kinetics human action video dataset. CoRR, 705.0005,,, 2017.
R. Khanna, N. Robinson, M. O'Donnell, H. Eyre, and E. Smith. Affective computing in psychother- apy. Advances Psychiatry and Behavioral Health, 2(1):95-105, Sep 2022.
D. Kollias, Multi-label compound expression recognition: C-expr database & network. In CVPR, 2023.
D. Kollias and S. Zafeiriou. Expression, affect, action unit recognition: Aff-wild2, multi-task learn- ing and areface. CoRR, 2019.
D. Kollias, Tzirakis, A. S. Cowen, S. Zafeiriou, Kotsia, E. Granger, M. Pedersoli, L. Bacon, A. Baird, C. Gagne, C. Shao, G. Hu, S. Belharbi, and M. H. Aslam. Advancements in Affective and Behavior Analysis: The 8th ABAW Workshop and Competition. 2025. doi: 10.6084/m9.
J. Kossaifi, R. Walecki, Y. Panagakis, J. Shen, M. Schmitt, F. Ringeval, J. Han, V. Pandit, A. Toisoul, B. Schuller, et al. Sewa db: A rich database for audio-visual emotion and sentiment research in the wild. TPAMI, 43(3):1022-10400 2019.
K. Kraack. A multimodal emotion recognition system: Integrating facial expressions, body move- ment, speech, and spoken language. CoRR, abs/2411.17907 2024.
Labbé. Colmegna, V. Valerio, V.G. Boucher, S. Peláez, A.I. Dragomir, C. Laurin, E.M. Hazel, S.L. Bacon, and K.L. Lavoie. Training physicians in motivational communication address influenza vaccine hesitation: a proof-of-concept study. Vaccines, 10(2):143, 2022.
I. Lee, E. Lee, and S.B. Yoo. Latent-ofer: Detect, mask, and reconstruct with latent vectors for occluded facial expression recognition. ICCV, 2023.
Li and W. Deng. Deep emotion transfer network for cross-database facial expression recognition. In ICPR, 2018.
S. Li, W. Deng, and J. Du. Reliable crowdsourcing and deep peeeeiippreserving learning for expres- sion recognition in the wild. In CVPR, 2017.
J. Liang, D. Hu, and Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In ICML, 2020.
B. Lin, Y. Ye, B. Zhu, J. Cui, M. Ning, P. Jin, and L. Yuan. Video-llava: Learning united visual representation by alignment before projection, 2024.
C. Liu, Zhang, X. Liu, T. Zhang, L.. Meng, Liu, Deng, and W. Jiang. Facial expression recognition based on multi-modal features for videos in the wild. In CVPR, 2023a.
D. Liu, H. Zhang, and P. Zhou. Video-based facial expression recognition using graph convolutional networks. In ICPR, 2021a.
H. Liu, R. An, Z. Zhang, B. Ma, W. Zhang, Y. Song. Y. Hu, W. Chen, and Y. Ding. Norface: Improving facial expression analysis by identity normalization. ECCV, 2024a.
X. Liu, L. Jin, X. Han, J. Lu, J. You, and L. Kong. Identity-aware facial expression recognition in compressed video. In ICPR, 2021b.
Y. Liu, W. Wang, Ch. Feng. H. Zhang, Z. Chen, and Y. Zhan. Expression snippet transformer for robust video-based facial expression recognition. Pattern Recognition, 138: 109368, 2023b.
13

---

Published as a conference paper at ICLR 2026
Y. Liu, Y. Zhang. and Y. Wang. Application of deep learning-based image processing in emotion recognition and psychological therapy. Traitement du Signal, (1())229, 2024b.
H. Lokhande, C. Garware, T. Kudale, and R. Kumar. Personalized well-being interventions (pwis): A new frontier in mental health. In Affective Computing for Social Good: Enhancing Well-being, Empathy, and Equity, pp. 183-200. 2024.
I. Loshchilov and F. Hutter. SGDR: stochastic gradient descent with warm restarts. In ICLR, 2017.
C. Luo, S. Song, W. Xie, L. Shen, and H. Gunes. Learning multi-dimensiona edge feature-based AU relation graph for facial action unit recognition. In IJCAI, 2022.
N.E. MacDonald. Vaccine hesitancy: Definition, scope and determinants. Vaccine, 33(34):4161- 4164, 2015.
Y. Maki, H. Yoshida, Yamaguchi, and H. Yamaguchi. Relative preservation of the recognition of positive facial expression "happiness" in alzheimer disease. International Psychogeriatrics, 25 (1):005-110, 2013.
J.K. Manuel and T.B. Moyers. The role of ambivalence behavior change. Addiction, 111(11): 1910-1912, November 2016.
J. Mao, R. Xu, X. Yin, Y. Chang, B. Nie, A. Huang, and Y. Wang. Poster++: A simpler and stronger facial expression recognition network. Pattern Recognition, pp. 1 110951, 2024.
H.P. McDonald, A.X. Garg, and R.B. Haynes. Interventions to enhance patient adherence to medi- cation prescriptions: scientific review. Jama, 288(22):2868-25 2879, 2002.
S. Michie, M. Richardson, M. Johnston, C. Abraham, J. Francis, W. Hardeman, M.P. Eceles, J. Cane, and C.E. Wood. The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interven- tions. Annals of behavioral medicine, 46(1):81-955 2013a.
S. Michie, West, and B. Spring. Moving from theory to practice and back in social and health psychology. 2013b.
W.R. Miller and G.S. Rose. Motivational interviewing and decisional balance: contrasting responses to client ambivalence. Behavioural and cognitive psychotherapy. 43(2):129-141) 2015.
J.A. Miranda Calero, L. Gutiérrez-Martin, E. Gtozerto-González E. Romero-Perales, J.M. Lanza- Gutiérrez, C. Peláez-Moreno, and C. López-Ongil Wemac: Women and emotion multi-modal affective computing dataset. Scientific data, 1 11(1):118) 2024.
A. Mollahosseini, B. Hassani, and M. H. Mahoor. Affectnet: A database for facial expression, valence, and arousal computing in the wild. IEEE Trans. Affect. Comput., 10(1):18- 31, 2019.
Murtaza, S. Belharbi, M. Pedersoli, and E. Granger. realistic protocol for evaluation of weakly supervised object localization. In WACV, 2025.
J. Nasimzada, J. Kleesick, K. Herrmann, A. Roitberg, and C. Seibold. Towards synthetic data gen- eration for improved pain recognition in videos under patient constraints. CoRR, abs/2409. 16382, 2024.
A. O'Donnell, M. Addison, Spencer, H. Zurhold, M. Rosenkranz, McGovem, Gilvarry, M.-S. Martens, U. Verthein, and E. Kaner. Which individual, social and environmental influences shape key phases in the amphetamine type stimulant use trajectory? a systematic narrative review and thematic synthesis of the qualitative literature. Addiction, 1 114(1):24 47, 2019.
C. Ortiz, T. López-Cuadrado A. Ayuso-Álvarez, C. Rodríguez-Blázquez, and I. Galán. Co- occurrence of behavioural risk factors for non-communicable diseases and mortality risk in spain: a population-bel -aaee cohort study. BMJ open, 15(1):e093037, 2025.
M. Pantic and L.J.M. Rothkrantz. Toward an sfffect-sensitive multimodal human comman- interac- tion. Proceedings of the IEFF, 1370 2003.
14

---

Published as a conference paper at ICLR 2026
L. Pepa. L. Spalazzi, M. Capecci, and M. G. Ceravolo. Automatic emotion recognition in clinical scenario: a systematic review of methods. IEEE Transactions on Affective Computing, 14(2): 1675-1695, 2021.
S. Poria, D. Hazarika, N. Majumder, G. Naik, E. Cambria, and R. Mihalcea. MELD: multimodal multi-part party dataset for emotion recognition in conversations. In Conference of the Association for Computational Linguistics, pp. 527-536, 2019.
R. G. Praveen and Ja. Alam. Inconsistency-awarre cross-attention for audio-visual fusion in dimen- sional emotion recognition. CoRR, 0tss/405..12855. 2024a.
R. G. Praveen, E. Granger, and P. Cardinal. Cross attentional audio-visual fusion for dimensional emotion recognition. In International Conference on Automatic Face and Gesture Recognition (FG 2021), 2021.
R. G. Praveen, P. Cardinal, and E. Granger. Audio-visual fusion for emotion recognition in the valence-arousal space using joint cross-attention. IEEE Transactions Biometrics, Behavior, and Identity Science, 5(3):360-373, 2023.
R.G. Praveen and J. Alam. Recursive joint cross-modal attention for multimodal fusion in dimen- sional emotion recognition. In ABAW workshop at CVPR, 2024b.
A. Radford, J. Wook Kim, T. Xu, G. Brockman, McLeavey, and I. Sutskever. Robust speech recognition via large-scale weak supervision. In ICLM, 2023.
N. Richet, S. Belharbi, H. Aslam, O. Zeeshan, A. L. Koerich, M. Pedersoli, S. Bacon, and E. Granger. Textualized and feature-based models for compound multimodal emotion recognition in the wild. In ABAW workshop ECCV, 2024.
F. Ringeval, A. Sonderegger, J. Sauer, and D. Lalanne. Introducing the recola multimodal corpus of remote collaborative and affective interactions. In International conference and workshops on automatic face and gesture recognition (FG), pp. 1-8, 2013.
E. Sánchez-Lozano, Tzimiropoulos, and Valstar. Joint action unit localisation and intensity estimation through heatmap regression. In BMVC, 2018.
A.V. Savchenko. Hsemotion team at abaw-8 competition: tional mimicry intensity and facial expression recognition. In ABAW workshop at CVPR, 2025.
Audiovisual ambivalence/hesitancy emo-
D. Sejdinovic, B. Sriperumbudur, A. Gretton, and K. Fukumizu. Equivalence of distance-basse based and rkhs-based statistics in hypothesis testing. The annals of statistics, pp. 2263-2291, 2013.
M. Sharafi, E. Ollivier, M.O. Zeeshan, S. Belharbi, M. Pedersoli, A.L. Koerich, S. Bacon, and E. Granger. Disentangled source-free personalization for facial expression recognition with neu- tral target data. In International Conference on Automatic Face and Gesture Recognition (FG), 2025.
M. Sharafi, S. Belharbi, H. Ben Salem, A. Etemad, A.L. Koerich, M. Pedersoli, S. Bacon, and E. Granger. Personalized feature translation for expression recognition: An efficient source-free domain adaptation method. In ICLR, 2026.
N.A. Sharma, A.S. Ali, and M.A. Kabir. A review of sentiment analysis: tasks, applications, and deep learning techniques. International journal of data science and analytics, 19(3):35-338, 2025.
M. H. Siddiqi. Ahmad, Alhwaiti, and F. Khan. Facial expression recognition for healthcare monitoring systems using neural random forest. IEEE Journal of Biomedical and Health Infor- matics, 2024.
S. Sinha, S.Kr. Mishra, and S. Bilgaiyan. Emotion analysis to provide counseling to students fighting from depression and anxicty by using cctv surveillance. In Machine Learning and Information Processing: Proceedings of ICMLIP 2019, pp. 81-94, 2020.
15

---

Published as a conference paper at ICLR 2026
B. Subramanian, J. Kim, M. Maray, and A. Paul. Digital twin model: A real-time emotion recogni- tion system for personalized healthcare. IEEE Access, 10:81155 10:81155-81165, 2022.
A. Suraj, A. S Kaushik, and K. Bai. Patient monitoring using emotion recognition. International Journal for Research in Applied Science and Engineering Technology, (999):119-12000 2022.
I. Sutskever, J. Martens, G. Dahl, and G. Hinton. On the importance of initialization and momentum in deep learning. In ICML, 2013.
A.D. Vairamani. Advancements in Multimodal Emotion Recognition: Integrating Facial Expressions and Physiological Signals, pp. 217-240. Cham, 2024.
M.J. Van Gent, M.C. Onwezen, R.J. Renes, and M. Handgraaf. Betwixt and between: A system- atic review on the role of ambivalence in environmental behaviours. Journal of Environmental Psychology, 97:102311, 2024.
B. Voisard, A.I. Dragomir, V.G. Boucher, G. Szezepanik, Simon L. Bacon, and K.I. Lavoie. Train- ing physicians in motivational communication: An integrated knowledge transfer study protocol. Health Psychology. 2024.
Waligora, Aslam, O. Zeeshan, S. Belharbi, A. L. Koerich, M. Pedersoli, Bacon, and E. Granger. Joint multimodal transformer for emotion recognition in the wild. In ABAW workshop at CVPR, 2024.
S. Walter, S. Gruss, H. Ehlciter, J. Tan, H.C. Traue. P. Werner, A. Al-Hamadi, S. Crawcour, A.O. Andrade, and Moreira da Silva. The biovid heat pain database data for the advancement and systematic validation of an automated pain recognition system. In International Conference on Cybernetics (CYBCO), pp. 128-131, 2013.
Wang and L. Kawka. Gimefive: Towards interpretable facial emotion classification. CoRR, abs/2402. 15662, 2024.
Y. Wang, S. Yan, Y. Liu, W. Song, Liu, Y. Chang, X. Mai, X. Hu, W. Zhang, and Gan. A survey on facial expression recognition of static and dynamic emotions. CoRR. abs/2408.15577 2024.
S.T. Wasim, M.U. Khattak, M. Naseer, S. Khan, M. Shah, and Fahad S. Khan. Video-focalnets: Spatio-temporaa focal modulation for video action recognition. In ICCV, 2023.
B. Williams. Practical human behaviour change for the health and welfare of animals. 2024.
7. Wu and J. Cui. La-net: Landmark-aware learning for reliable facial expression recognition under label noise. In ICCV, 2023.
Xue, Q. Wang, and G. Guo. Transfer: Learning relation-aware facial expression representations with transformers. In ICCV, 2021.
Xue, Q Wang, Z. Tan, Ma, and G. Guo. Vision transformer with attentive pooling for robust facial expression recognition. IEEE Transactions Affective Computing, 2022.
J. Yan, P. Li, C. Du, K. Zhu, X. Zhou, Y. Liu, and J. Wei. Multimodal emotion recognition based facial expressions, speech, and body gestures. Electronics (2079-9292), 13(18), 2024.
S. Yang, J. Van de Weijer, L. Herranz, and S. Jui. Exploiting the intrinsic neighborhood structure for source-free domain adaptation. NeurIPS, 2021.
M.O. Zeeshan, M.H. Aslam, S. Belharbi, A.I. Koerich, M. Pedersoli, S. Bacon, and E. Granger. Subject-based domain adaptation for facial expression recognition. In International Conference on Automatic Face and Gesture Recognition (FG), 2024.
M.O. Zeeshan, M. Pedersoli, A.I. Koerich, and F. Grange. Progressive multi-source domain adap- tation for personalized facial expression recognition. TFFC, pp. 1-12, 2025.
D. Zeng, Z. Lin, X. Yan, Y. Liu, F. Wang. and B. Tang. Face2exp: Combating data biases for facial expression recognition. In CVPR, 2022.
16

---

Published as a conference paper at ICLR 2026
C. Zhang, H. Liu, Y. Deng. B. Xie, and Y. Li. Tokenhpe: Learning orientation tokens for efficient head pose estimation via transformers. In CVPR, 2023a.
S. Zhang, Z. Zhao, and C. Guan, Multimodal continuous emotion recognition: A technical report for abaw5. In ABAW workshop CVPR, 2023b.
Y. Zhang. R. Zhao, W. Dong, B.-G. Hu, and Q. Ji. Bilateral ordinal relevance multi-instance regres- sion for facial action unit intensity estimation. In CVPR, 2018.
S. Zhao, G. Jia, J. Yang. G. Ding, and K. Keutzer. Emotion recognition from multiple modalities: Fundamentals and mothodologies. IEEE Signal Processing Magazine, 38(6):59-73, 2021.
C. Zheng, M. Mendieta, and C. Chen. POSTER: A pyramid cross-fusion transformer network for facial expression recognition. In ICCVw, 2023.
17

---

Published as a conference paper at ICLR 2026
Appendix
Table of Contents
Recommendations for Future Works in A/H Recognition
18
B Related Work
18
BAH Dataset Limitations
21
D BAH Dataset File Structure
22
E BAH Data Collection Web-platform
22
BAH Dataset Diversity
23
G Demographics Analysis of Train, Validation, and Test Sets
30
H Analysis of Annotators Cues
33
BAH Dataset Annotation Codebook
35
Baseline Results
41
Visual Modality: Cropped Faces vs Full Frame Head Head-pose pose Features J.1
41
J.2
Supervised Learning Cases (Followup from Main paper)
41
J.3
Zero-shot Inference: Multimodal Large Language Models (M-LLMs)
43
J.4
Personalization using Domain Adaptation
45
A RECOMMENDATIONS FOR FUTURE WORKS IN A/H RECOGNITION
The newly introduced A/H recognition task requires the model to be able to detect affect conflict cross-modalities in addition to within-modality. Standard multimodal models are trained yield predictions aligned with the output supervision. This automatic training may focus on learning label patterns and miss acquiring mechanism to understand affect conflict. To build more inter- pretable A/H recognition systems, we recommend 2-level framework. The first level should focus on modelling affect per modality in an independent way. Off-the-shelf pretrained sentiment anal- ysis (Sharma et al.] 2025) models could be used. This first level should be separated from A/H since we can not detect at modality level yet, at least for the cross-modality case. At the see- ond level, dedicated fusion mechanism should be used to assess whether there is affect conflict cross cross-modand modalities to make a decision. This module does a deeper work than simply fusing features commonly done. should acquire understanding of affect conflict to be able to detect it. Such modular and interpretable framework allows introducing priors about affect conflicts.
Statistics extracted from annotators could be leveraged to constrain the model find conflicts cross-modalities. A specialized temporal modelling should be used to detect within-modality con- flict based on the output of the first level. Statistics from A/H segments durations should be consid- ered as A/H happens briefly. Context is also important to detect within--odalle cases. Segmented body could help as well since it contains important cues and less noise compared full frame. Our results showed that standard multimodal models and fusion technique yield modest performance. Future works should focus on designing specialized frameworks for A/H recognition.
B RELATED WORK
This section provides works in affective computing related to behavioural science.
18

---

Published as a conference paper at ICLR 2026
a) Affect Recognition using Machine Learning:
Basic Emotions. An important line for ML research in affective computing is discrete emotion
recognition in facial image (facial Expression Recognition FER) (Bonnard et al.) [022] Liu al.
2024a; Kollias et al. 2025 Lee et al. [023 Mao et al. 2024 Wang et al. 2024| Wu & Cui 2023
Xue et al.. 2021 Zeng et al.] 2022] Zheng al. 2023). This usually involves classifying facial images into one of seven or eight basic emotions, such as 'Happy', 'Sad', and 'Surrprsed'.. Other works focus on videos (Liu et al.] [023a] 2021a b 2023b) as well. There has also recent interest in designing robust FER methods that are interpretable (Belharbi et al.] 2024a Wang & Kawka, 2024] Xue et al., 2022). They typically produce heat map that points to relevant regions used by model to perform prediction. This is usually formulated as an attention map or Class-Activalee Map (CAM) (Choe et al.] 2022, Murtaza et al., 2025). Other work aims predict ordinal levels (i.e, ordered labels), including pain and stress estimation (Aslam et al. 2024, Chaptoukaev et al.) 2023 ZZeshan et al. 2024 Nasimzada et | (202);; a task that can be extremely useful in healthcare applications. Some datasets such as BioVid d (Walter et al., 2013) rely on advanced and expensive modalities such as bio-signals to predict pain for instance. Dimension recognition of emotions, typically aims to estimate continuous valence and arousal values linked to emotions (Dong et al.) 2024 Praveen&& & Alam 2024at Praveen et al. [023] (021). Finally, another task related in emotion recognition is Action Units (AUs) detection (Jacob & Stenger [202]] Luo et al., 2022). It aims to predicting active AUS in the face under a multi-label classification framework. Other works go further to estimate the intensity of AUs (Fan et 2020: Zhang et al. 2018). or both (Sánchez-
Lozano et al. 2018), much more challenging task.
Compound Emotions. Real-world scenarios often present complex emotions that combine basic ones. There has been recent interest in building affective computing models to predict compound emotions, a case where a mixture of basic emotions are expressed [Kollias) 2023 Richet et al. 2024). These are show in several practical real-world application since such complex emotions oc- cur daily interactions. However, they are more difficult to recognize as they are subtle, ambiguous, and resemble basic emotions. A recent specialized video-based dataset named C-EXPR-DE (Kol-) lias| 2023) has been constructed for the design/cvaluation of models. The dataset accounts for the difficulty of the task different modalities are required to better recognize compound emotions.
Despite the recent progress in affect modelling, Ambivalencee/esiiannyy recognition is still unex- plored in ML. A possible reason is the lack of specialized dataset for training and evaluation of ML models. As is implicated in healthcare and interventions, A/H is a common topic in be- havioural science (Hohman al. 2016 Manuel & Moyers 2016). A/H recognition is related to compound emotion recognition task where intention and attitudes are conflicted or in a in-hetween state, between willingness and resistance (MacDonaid) 2015), or positive and negative affect (Ar- mitage & Conner 2000). This can manifest in how an individual expresses them self and can be
Dotnuct
Alfect
Siataliio
Sulject- hundd
Nuur. partiiipura
Numa of sasaplo
Earinnssnct
Ausslation
83P-D6 de 2017
staricts
Teuges
15,30 image
widl
Traig leail
Hark rrasiiia
Tragas
ISE images
Irage lasal
AEE
Kallia& & Zaciioo2001
2019
VidacttAreal
Vides sadio
564 video
Wid
Frssslerd
Actiri lat
Orepuia crsdion
Viko,
tideo
Prareetod
Firre, Himelli aggals
Biooid
3113
Pai cairaiio
(GaR) BCG,
90
19007 sumples
Lab
Franoledd
RC00GA
Rayora al 2003
Aquare Excetvaal
Recsption
plyillagy iekervend eedignon elstindomnal
videas
Frane
CDRA
2015
Aquuran Enstival
Recipition
Vidi zudio
Yeu
300
1,00 valoe
WQJ
Prane bool
KEMAC
Minada Calers ct 2014
Dicaste diseddeal metion
Hasickoyy (Maad vilane polse.
galratt das requensc
Yes
100
300 rocads
Lab
Sc2-csporid
EDA
Stzeante
Chanssaar a 23C3
Stro4
Reppraton, Faee vider, Speedh
Yes
65
587 vidoos
Lab
Prane-ted
butzetiee
SchoMac
Miday dtal 2015
Spectoeuse
video
Yos
91
Pt videos
Wid
Viooi
3334
Cancnalie
Suppon
vidan zedio Met
Yeu
1,10 dalagao
Wid
Ulannaslord
of
at 200e
sociatia for
vidan sedio, Met
Yo
a0
Porastewl
BMB bo00
Video sertio Lnancript
Yes
1,27 viloos
Foreo linel
Table 7: Common affective computing datasets for emotion modelling in health contexts.
19

---

Published as a conference paper at ICLR 2026
recognized (Hayashi et al. 2023) in their facial expression, tone, verbal, and body language. As result, A/H exhibits a mulumodal nature that comes as the result of subtle interconnection between different cues. Unfortunately, such discord is extremely difficult to spot; a task that requires human training. This is a tedious and expensive procedure, leading to ineffective and less scalable eHealth interventions under limited resources. Assisting healthcare providers with automatic, reliable and inconspicuous tools to help them recognize A/H can have a major impact in improving eHealth interventions.
Our BAH dataset fills in the gap in the literature, and to provide an important resource to de- sign/evaluate ML models for A/H recognition task. is a video Q&A dataset from which we extract audiovisual information with transcripts, offering multiple modalities. The dataset is fully anno- tated by behaviour science experts at video- and frame-level. to recognize A/H at each segment are provided. This includes facial and vocal expressions, body language, language in addition to highlighting where there inconsistency between the modali- ties. As shown in Table 7 our BAH dataset is competitive compared to existing affective computing datasets in terms of modalities, number and diversity of participants, and annotations. While no dataset matches the specific focus on A/H in digital health interventions, datasets like MESC (Chu jet al.) 2024), SchiNet (Bishay et al. 2019), and IEMOCAP (Busso et al.] (2008) contain videos from interviews with psychological relevance. Therefore, BAH provides important asset for the MI. community to begin research in A/H recognition.
addition, cues used by annotators
b) Behavioural Science:
Health-Related Behaviour Change and Non-Communicable Diseases. High-risk health be- haviours, such as tobacco use, physical inactivity, unhealthy diets, and harmful alcohol consumption, are responsible for the vast majority of non-communicable diseases (NCDs), which include cardio- vascular disease, type 2 diabetes, cancer, and chronic respiratory illnesses. According to the World Health Organization (WHO) (Ortiz et al. (2025). NCDs account for approximately 74% of global deaths, and these outcomes are disproportoonately influenced by modifiable behavioural factors. Ev- idence suggests that around 80% of chronic disease risk is attributable to these high-risk behaviours.
Consequently, health-related behaviour change has become a primary target for preventive and ther- apeutic interventions. Traditional methods, such as motivational interviewing (MI) and cognitive behavioural therapy (CBT), rely on face-to-face clinical interviews, which remain foundational to behavioural health practice (O'Donnell et al.) 2019). These interactions provide unique opportuni- ties for clinicians to detect ambivalence, hesitancy, and other complex affective states, often through subtle verbal and nonverbal cues (Hall et al.) [1955). Despite the growing shift toward digital plat- forms, clinical interviews remain the gold standard for eliciting meaningful emotional and cognitive responses, insights that essential to tailoring behaviour change strategies. Efforts to change health behaviours over the long term are inherently complex. Individuals often experience ambiva- lence and hesitancy, understood fluctuating between intention and resistance, when attempting to adopt healthier lifestyles. In traditional healthcare contexts, providers rely on both verbal com- munication and non-verbal cues (e.g.. tone, gestures, facial expressions) to recognize and address such motivational conflicts. This in- in-person interaction allows for nuanced support that can adapt to a patient's 's readiness for change (Davidson & Scholz, 2020p. The purpose of developing multimodal A/H recognition systems is to capture and replicate this nuanced understanding of patient behaviour within digital health interventions, thereby supporting clinicians and scaling behavioural health care.
Multimodal Cues and the Detection of Complex Emotions. Identifying complex emotional states such as ambivalence, resistance, or hesitancy is crucial for tailoring behavioural interventions. Re- search in psychology and human-computer interaction has shown that complex emotional states, such as ambivalence, uncertainty, or defensiveness, are communicated through a combination of fa- cial expressions, body posture, vocal tone, speech patterns, and physiological responses (Guo al. 2018 Pantic & Rothkrantz 2003). In digital contexts, however, the absence of physical presence makes this task more difficult. Recent research in psychology and computer science has focused on the use of multimodal cues, such as facial expressions, voice tone, body posture, and physiological responses, as proxies for emotional and motivational states (Kraack [2024] Yan al. 2024). These cues can reveal underlying emotional conflict or uncertainty that might not be captured by self- report alone. Studies have shown that combining multiple input channels (e.g., audio-visual data) can enhance the accuracy of emotion recognition systems. For instance, multimodal datasets are be- ing used to train models that detect affective states like confusion, frustration, and mixed emotions,
20

---

Published as a conference paper at ICLR 2026
which highly relevant in contexts such education, mental health, and behaviour change. By incorporating these data streams, researchers can better approximate the nuanced human capacity for reading emotions, paving the way for emotionally aware systems (He et al. 2020: Zhao et al. 2021).
Affective Computing and Personalized Digital Health Interventions Affective computing, sub- field of artificial intelligence (AI) focused on recognizing, interpreting, and responding to human emotions, holds promise for advancing personalized digital health interventions. By leveraging emotion-aware algorithms, digital platforms can better understand users' psychological readiness and tailor support accordingly (Lokhande al. 2024 Vairamami] 2024). For example, interventions that dynamically respond to detected signs of resistance or disengagement may improve user reten- tion and behavioural outcomes. Incorporating affective computing into digital health technologies also allows dynamic tailoring of content based on users' real-time affect, responsive dialogue, mim- icking the adaptability found in face-to-face interactions. Recent advancements conversational agents, voice analysis, and facial expression recognition have made it possible for digital interven- tions to adapt content delivery based on real-time emotional assessments (Khanna et al., (0022). This not only improves user engagement but also enhances intervention effectiveness by ensuring mes- sages are delivered in an emotionally congruent and contextually appropriate manner (Hornstein) al. 2023
C BAH DATASET LIMITATIONS
While BAH dataset offers novel contribution to emotion recognition for digital behaviour change, several limitations should considered.
1) Data collection constraints. The web-based platform occasionally experienced technical issues, preventing some participants from completing seven videos. As participants used their own devices in home settings, video and audio quality varied significantly despite clear instructions and testing. Response length was participant-determinng leading to high variability in content. Some environmental noise visual distractions (e.g., background conversations, movement) were present in subset of recordings.
2) Participant representation. Although participants recruited from nine Canadian provinces with diverse age and ethnic backgrounds, individuals from under-resourced areas or without reliable internet access were likely underrepresented. Gender identity was collected but used in sampling, and no data on socioeconomic status was recorded. Digital literacy and access may have biased participation toward more tech-savvy individuals.
Frames
split
split-frames transcription
Videos
BAH dataset documentation.. pdf
BAH Dataset EULA-2 2. pdf
bah-video. CSv
extract frames from videos. py
extract frames_from from fromees_room_vidos... videos. sh meta data.ymi readme md version. txt
video annotation transcript. yaml
Figure 4: File structure of the shared BAH dataset.
21

---

Published as a conference paper at ICLR 2026
3) Multimodal and data balance issues. The expressiveness of cues (facial, vocal, bodily, verbal) varied widely by participant, complicating consistent multimodal analysis. Though the dataset is balanced at the video level, frame-level imbalance exists (fewer A/H frames than non-A/H). Training strategies that account for class imbalance should be considered.
D
BAH DATASET FILE STRUCTURE
Figure
shows the file structure the shared BAH dataset. The file
"BAH_ dataset "BAHseattset dww.aaee_douuueeeeeeeeeeeeeeeeee www.mmntationn.poenoo pdf" contains the detailed documentation about all files/directory,
including annotation structure.
BAH DATA COLLECTION E
Alongside the dataset files, we include a slide presentation of our "Automatic Expression Recog- nition" (AER) web-based platform (www.aersudyy.ca) The presentation in the file "AER-web- platform.pdff Figure shows an example of the platform.
22

---

Published as a conference paper at ICLR 2026
Recording test
Please count
any
You have good
lighenn
You are cencored
You are facing the
everything looks good
camera
Continue
Your voice is clear
cas the tuuss persiting plaas sontec at:
Choose who will be talking to:
Claire
Flisa
Question 1
While answering the previous question would describe
predornnnatt emotion neutral (neither positive nor negative)
Strongly chsggeee
Douggee
agree diaapro
Agea
Shorglvagee
ponter
Continue
Figure 5: Examples taken from the platform present "Automatic Expression Recognition"
(AER) web-based platform (ww..aerstudy.ca)
F BAH DATASET DIVERSITY
This section includes statistics about BAH dataset to highlight its diversity. Figure 11 shows a general overview via a nutrition label. Overall, BAH dataset has significant diversity. It covers different Canadian provinces, age range, ethnicities, and male/female presence. It has large number
23

---

Published as a conference paper at ICLR 2026
of videos (1,427) where 778 videos contain A/H. Most asked questions elicited A/H, especially question-4 (Ambivalent). addition, since we have less control over the participants, and their environment, the dataset is considered in-the-wild. On top of video and audio modality, we provide audio transcript which has shown to be an important modality for A/H recognition. BAH is fully annotated at video- and frame-level. Moreover, annotators report the used cues to recognize A/H at each segment. All these properties make our dataset a realistic and relevant asset to design ML model for the task of A/H recognition in videos.
We include the following general information:
Videos durations distribution (Figure 6a
Videos per participants distribution. (Figure 6b
Questions and A/H distribution (Figure 6c)
A/H segments duration (Figure
Participants sex distribution (Figure 6d
Histogram of videos duration (sec)
500
400
Freo enc)
Frequ enC)
300
200
100
Video duration (sec)
Number of videos participant
(a) Videos duration histogram.
(b) Distribution of number of videos per participant.
20M
52.0
47.3
15.4
13.5
13.6
12.8
Froquency (%)
25
6.
033
0.8
84
07
Question
finak
Male Gender (All datal
Intee serooay
(c) Distribution over questions: Num. videos per- question (blue), Num. videos with A/l (red).
(d) Sex distribution.
Figure 6: Video duration a), and videos/participant bb. question distribution (c). and distribu- tion d over BAH dataset.
24

---

Published as a conference paper at ICLR 2026
Histogram of AM segments duration (sec)
Histogram of A/H segments duration (frames)
500
Irequency
Requency
100
10
1.00
200
300
500
600
A/H segment duration (sec)
A/H segment duration (trames)
(a) Distribution of A/H segment duration in seconds.
(b) Distribution of A/H segment duration in frames.
Figure 7: Distribution of A/H segment duration seconds a and frames
b over BAH dataset.
addition, more demographics statisties are included as well:
Participants' age distribution (Figure g).
Participants' age range distribution (Figure 9a).
Distribution of Canada provinces where participants live (Figure 9b).
Participants' simplified ethnicity distribution (Figure 10ap.
Participants' student-status distribution (Figure 10b).
Participants' consent to their data in challenges distribution (Figure 10c].
Participants' consent to their data in publications distribution (Figure 10d
Participants' birth country distribution (Table
25

---

Published as a conference paper at ICLR 2026
Frequency (%)
Age (All data)
Figure 8: Participants' age distribution in BAH dataset.
Frequency (%)
frequency (%
20.7
10.0
province data)
75. Mileee
peir Age range (All data)
pwrs Oee
(b) Distribution of Canada provinces where partici- pants live.
(a) Participants' age range distribution.
Figure 9: Participants' age range (a). and where the provinces where they live (b) over BAH dataset. Name of provinces: 'Manitoba (MB)', 'Alberta (AB)', "Nova Scotia (NS)', "Newfoundland and Labrador (NL)'. "Saskatchewan (SK), 'New Brunswick (NB)', "Ontario (ON)', 'Quebec (QC)',
"British Columbia (BC)'.
26

---

Published as a conference paper at ICLR 2026
54.0
67.0
Frequency 155
27,3
10.7
5.7
0.3
Ethnicity simplifled (All data)
Is student (All data)
(a) Participants' simplified ethnicity distribution in (b) Participants'
student-status distribution in BAH
BAH dataset.
datasct.
97.7
79.7
Frequency 196
Frequency (96)
2.3
Use challenges (All dota)
Use in publications (All datal
(c) Participants' consent to use their data in challenges distribution in HAH dataset.
(d) Participants' consent to use their data in publica- tions distribution in BAH dataset.
Figure 10: Distribution of participants' simplified ethnicity (a). their student-status (b). their consent their data in challenges (c) and publications (cl over RAH dataset.
27

---

Published as a conference paper at ICLR 2026
Birth country
Number of participants
Algeria
Australia
Bangladesh
2
Belize
Bulgaria
1
Canada
127
China
8
Colombia
France
Germany
Ghana
Hong Kong
India
5
Japan
Kenya
Macedonia
New Zealand
Nigeria
Null
Peru
Philippines
6
Russian Federation
Saint Lucia
South Africa
Sri Lanka
2
Taiwan
1
Thailand
Tunisia
1
Turkey
2
United Arab Emirates
1
United Kingdom
6
United States
3
Vietnam
1
Table 8: Distribution of participants' birth country in BAH dataset. Top-3 countires are in bold.
28

---

Published as a conference paper at ICLR 2026
BAH Dataset Facts
Dataset BAF (Behavioural Amblivalence/Hessiannyy - A/H)
Nature of Dataset A Dataset for Ambivalencce/Hesttanny recognition in
videos for participants recruited in Canada Participants Country Canada
Number of provinces in Canada 9
Provinces Canada 'Manitoba (MB), 'Alberta (AB)' 'Nova Scotia (NS),
Newfoundland and Labrador (NL)', 'Saskatchewan (SK), 'New Brunswick
(NB)', 'Ontario (ON)', 'Quebec (QC)', 'British Columbia (BC).
Number of participants 300
Number of videos 1, 1,427 where 778 videos contains A/H
Average video length 26.76 + 16.47 (seconds) with a minimum and maximum duration of 3 and 96 seconds
Total duration 10. 60 hours where A/H duration is 1.79 hours
Total number of frames 916,618 where 156, 156,255 frames contains A/H Total number of A/H video segments 1,504
Length A/H video segment 1.29 + 2.45 seconds or 103.89 + 58.70
frames. The minimum and maximum A/H segment is 0.004 seconds (1
frame), and 23.8 8 seconds (572 frames).
Data capture web-platform www.aerstudy.ca
Motivation
Behavioural Ambivalence//esiitann (2AH) is a dataset collected for participant-basse multimodal recognition of A/H in videos. It contains videos from 300 participants captured across provinces Canada, with different age, and ethnicity. Through our web platform, we recruited participants to answer questions, some of which were designed to elicit A/H while recording themselves via webcam with microphone. BAH amounts to 1,427 videos for total duration of 10.60 hours with 1.79 hours of A/H. Our behavioural team annotated timestamp segments to indicate where A/H occurs, and provide frame- and video- level annotations with the A/H cues. Video transcripts and their timestamps are also included, along with cropped and aligned faces in each frame, and variety of participants meta-data. Original Authors
Summary
Manuela González-González, Soufiane Belharbi, Muhammad Osama Zeeshan, Masoumeh Sharafi, Muhammad Haseeb Aslam, Marco Pedersoli, Alessandro Lameiras Koerich, Simon L.
Bacon, Eric Granger
Metadata
URL
dataset eHealth, digital health intervention,
Keywords Ambivaaenneehhesitnnyy
video, Deep Learning, Benchmark
Available participants meta-data
Age, birth country, Canada province where the participant lives, ethnicity, ethnicity simplified, sex, student status, consent to use recordings in publications
Video format
mpp
Ethical Review Concordia University under agreement nº 30019002 and
ETS under agreement H2023120S
License
Custom for research purposes only. How to request the data? Fill in this form, sign, and upload the EULA
https:/www.cchscc.ca/rreees
First release
2025
Annotation
experts in behavioural science Video- and frame-level Label "1" for presence of A/H, "0", its absence Cues provided by annotators for each A/H segment Facial expressions, body language, audio and language addition highlighting where there is inconsistency between the modalities
Annotators
Data size
All files zipped (".zip)
9.6
Figure 11: A data card styled (nutrition label) for BAH dataset.
29

---

Published as a conference paper at ICLR 2026
G DEMOGRAPHICS ANALYSIS OF TRAIN, VALIDATION, AND TEST SETS
We provide in Figure 12 and 13 distributions of the splits train (195 participants), validation (30 participants) and test (75 participants) in BAH dataset. The split participants-base where videos of participant belong exclusively to one split. We ensure that all splits have similar distribu- tion in terms of main factors that are deemed important A/H variation across the population: gender, age, and ethnicity. We also include provinces distribution in Figure 14
Frequency (%)
Frequency (80
24,1
9.7
1.0
4i vesn Age range (Trainset data)
25 -trosannn
over yo
Age (Trainset dota)
(a) Train set split participants age distribution.
(b) Train set split participants age-range distribution.
36.7
Frequency (%)
Frequency 196
20.0
10.0
(Validset data)
(d) Validation set split participants age-range distribu- tion.
(c) Validation split participants age distribution.
frequency (%)
Peequeccy (%)
dllivin
Age (lestset datal
Age range (Testset data!
(e) Test set split participants age distribution.
(i) test set split participants age-range distribution.
Figure 12: Participants age and age-range distribution across all splits (train, validation, and test) in BAH dataset.
30

---

Published as a conference paper at ICLR 2026
56.4
45.
Frquency (%5)
Frequency 0%)
20.0
0,2
0.5
0.5
Ethnicity simplified (Trainset datal
Gender (TFeinsst
(b) Train split participants simplified ethnicity dis- tribution.
(a) Train set split participants sex distribution.
53.3
46.7
Fecuency (%2
Trequency (5
13.3
13.3
10.0
Ethnicity simplified (Validset data)
Gender (Volldset data)
(d) Validation set split participants simplified ethnicity distribution.
(c) Validation split participants distribution.
50.7
Frequency (56)
26.7
frequency (%6)
12.0
Must
Wnle
Ethnicity simplified llestset data)
Fanide
Pete lerle
Gender (Tastset datal
(f) test set split participants simplified ethnicity distri- bution.
(c) Test set split participants sex distribution.
Figure 13: Participants sex and simplified ethnicity distribution across all splits (train, validation, and test) in BAH dataset.
31

---

Published as a conference paper at ICLR 2026
26.2
26.7
26,7
Frequency (%
Frequency (S6)
3.6
Canada (Trainset
Canada (Vaildset
(a) Train set split participants provinces distribution.
(b) Valid set split participants provinces distribution.
16
2n
Canada province (Testset data)
(c) Test set split participants provinces distribution.
Figure 14: Participants provinces distribution across all splits (train, validation, and test) in BAH dataset.
32

---

Published as a conference paper at ICLR 2026
H ANALYSIS OF ANNOTATORS CUES
Figure 15a shows the distribution of modalities used by annotators to assess the presence of A/H. Overall, the four modalities contribution almost at the same rate in detecting A/H. Facial lead, followed by language, then body and audio. This suggests that the four modalities are equally important for A/H recognition. When looking to co-occurrence of these cues, we find that the four cues at once dominates by 54.1%, followed by facial-language" This suggest an adapted fusion of these four modalities is necessary be able to assess there is an affect between them.
with 14.7%, and "audio-
We include in Figure T6 17 distribution of cues used per modality. It shows that "Pause" the dominate cues used in audio modality; "Filler sound" for language; "Gaze" for facial; and "Shake" for body.
The inconsistencies between facial and language cues seem the lead by 42.6% of cases followed by language and body with 26.6%. This could be used priors recognize the case of A/H cross-modalities. When looking to co-occurrence, similarly, facial and language dominates alone with 21.4% of cases, followed by facial-language and language-body with the using them as constraints could help designing better A/H frameworks.
We also analyzed the inconsistencies between modalities in Figure 88a 18b
rate. Leveraging these statistics by
A/H also be present within single modality alone making it detection much more difficult compared to the case of cross-modality. Figure 19a shows that a large part of A/H cases fall into the case of within-modality, which is also spread across all questions (Figure 19bp. A dedicated per-modality temporal modelling could help detect these cases.
These aforementioned statistics show that recognizing A/H requires considering all four modalities, and being able to detect affect conflict across pairs and higher combinations of modalities for the case of cross-modality. Additionally, better temporal modelling required detect A/H in the case of within modality.
25.2
23.7
22.7
Frequenccy 2%
Cues (All data)
Cues co-occurence (All data)
(a) Modalities cues distribution.
(b) Modalities cues co-occurrence distribution.
Figure 15: Modalities distribution used by annotators BAH dataset.
33

---

Published as a conference paper at ICLR 2026
Frequency 0%
Frequercy
13
Audio Cue (All data)
Language cue (AlI datal
(a) Audio modality cues distribution.
(b) Language modality cues distribution.
frequency (60
Feequancy
13
data)
Body cue
(c) Facial modality cues distribution.
(d) Body modality cues distribution.
Figure 16: Per-modality cues distribution used by annotators in BAH dataset.
Stuttering
Slw
Click
No mot Hedging
Positive Fail
Cap
Volume
Repetition
Pause
Shaky voice Cut
words
Filler sound
Filler
sound
Excuse Correction
No opp
Negative
No cap
Mot
Breath Laugh Fas
lepetise
t
Filler word
Success
(a) Word cloud of audio modality cues.
(b) Word cloud of language modality cues.
Smile e Mouth
Open
Hangingfyyerows
Look
away Res lless
Lip
press
Shruge
80
Thr ow
snrow
Sigh
Shake
Nod
Scratch
Close
t
Eye
roll
OW Wrinkle chin close
Pout
Hands
Frown
2
Squint
(c)
Word cloud of facial modality
(d) Word cloud of body modality
Figure 17: Word cloud of per-modality cues used by annotators in BAH dataset.
34

---

Published as a conference paper at ICLR 2026
Frequency 196
Cues inconsistencies (All data)
Cues incondistencies co-occuraacees (Ali data)
(a) Cues inconsistencies distribution.
(b) Cues inconsistencies ro-occurrence distribution.
Figure 18: Cues inconsistencies and their co-occurrence distribution used by annotators in BAH dataset.
308
90
Number of segments
Number segmenns
0
19
Number incondistencies [Ali data)
with and without
65 (b) Cues distribution with without cross- modality inconsistencies over questions.
(a) Cues distribution with without coss-modalltty inconsistencies over segments.
Figure 19:
Distribution of cues with and without cross-modality inconsistencies segments; in addition to question-basee distribution used by annotators in BAH dataset.
I BAH DATASET ANNOTATION CODEBOOK
This section contains relevant information regarding designed annotation codebook for A/H recognition. We provide the definitions of A/H and the types of cues (Table 99), as well as a more detailed description of the relevant cues in each modality used detect A/H, which include fa- cial cues (Table IO), language cues (Table TT). audio cues (Table body language cues (Table [3), and cross-modal inconsistency cues (Table The codebook is working document that continues to evolve in response to relevant insights emerging from expert annotations and contributions from behaviour change experts. Updates on the codebook will be made available and communicated upon request. This iterative approach aligns with established qualitative research practices, where coding frameworks are refined throughout the analysis process better reflect the complexity and richness of the data (Bradley et al. 2007).
35

---

Published as a conference paper at ICLR 2026
Term Ambivalence/Hesitanny
Definition
The simultaneous presence of competing positive and negative feelings, ideas, thoughts, or emotions towards one same ob- ject or goal. state in which a person has entirely made up their mind about doing something; when they aren't fully decided on how act (towards behaviour or object; not nec- essarily the goal behaviour; excluding towards language or swering questions)
Facial Cues
Different motions of the muscles in the face. Facial expres- sions commonly occur around the mouth and eyes, including changes in a person's gaze. They can be used to assess a per- son's 's emotional state.
Language Cues
Includes verbal/speech-basse expressions of ambivalence or hesitancy. Some common verbal expressions can include the of 'I want to... but..." 'mmmm' among others.
Audio Cues
Changes a person's non-verbal language, such as changes in tone, speed and pitch.
Body Cues
Non-verbal signals that include gestures, body posture and movements. Some of the cues that can be annotated as body language are hand movements, head tilts, shoulders shrugging and sighs (chest movement).
Cross-modal inconsistency Cues
Simultancous incompatibility between two or more modalities different types of cues. For example, this could be repre- sented by someone saying 'yes' while shaking their head side side.
Table 9: BAH dataset annotation codebook: definitions.
36

---

Published as a conference paper at ICLR 2026
Facial cues
Definition
Upper Region
Close 2
A change in the frequency with which someone blinks closing one's me's eyes for longer period (e.g., either keeping them closed, or blinking for a long time). This excludes normal blinking, it is annotated when there is a difference compared to the participants baseline. Closing of both eyes; "blinking" (with both eyes).
Close
Closing one eye at the time; includes winking. The duration of the wink is not relevant, can be a quick wink or a longer one.
Squint
Partially closing one both eyes. Significant or identifiable changes or con- tractions in the muscles around the outer or inner corners of the eyes. It might involve some changes in the eyebrows, forehead and cheeks. Includes squinting eyes, muscles contracting around the eyes.
Frown
To bring your eyebrows together (inner eyebrow) so that there are lines on your face above your eyes. Frown, forehead fold, small frown, tensed forehead, wrin- kled forehead, furrowing brows, lowering inside corners of eyebrow
Eyebrow
Lowering/raising external parts of eyebrow(s) (or full eyebrow(s)). [i.e., both eyebrows]
Gaze
Changes in the direction of the gaze by moving the eyes. Moving eyes (not face) to look down, up, to the side.
Eye roll
Eye-rolling a transitory gesture in which person briefly turns their eyes up- ward, often in an arcing motion from one side to the other. The eyes do not set on anything in particular and go back to their previous position.
Open
Opening the eyes, looks like increase in awareness. Eyes look slightly bigger. Engagement of the eyelids, contracting the eyelid muscles to make them look wider.
Lower Region
Smile
Ends of the mouth/lips curve up, often with the lips moving apart. Includes: Smile, smirking, half a smirk, fake smile, raising both sides of the mouth, side smile, half smile
Pout
Pushing one's lips one's bottom lip forward; turning the outer sides of the lips downwards. Pouting, pursed lips, "frowning" with one's 's mouth Contracting or pressing lips without pushing them forward. Includes: pressed lips, pressing lips together, putting lips together. Excludes pressing lips to pout/purse.
Lip press press
Hanging
Leaving one's mouth open for an extended time (e.g., hanging mouth, gaping mouth).
Mouth
Any other movements of the mouth that (1) are not captured by smile, pout, or pressing lips, and (2) is not result of the baseline speech patterns. Opening mouth, opened mouth, raising upper lip, rising one side of the mouth, taking the mouth corners back and lower them
Wrinkle Chin
Moving the muscles around the chin to create identifiable lines, folds, ridges or furrows in the chin. Usually seen as a contraction of the chin muscles creating creases around or the chin.
Nose
Changes in the movement or looks of the nose. Includes significant movements on the nostrils, the tip of the nose, scrunching the nose, or any other muscle movement that would create change in the nose.
Table 10: BAH dataset annotation codebook: facial cues.
37

---

Words used that do not contain substantive content, but are used as fillers to fill in space while the person thinks (or to signal they not done talking. Published as a conference paper at ICLR 2026
Language cues
Definition
Filler sound
Sound made during pause in speech signalling the person isn' done taking. Examples: "mmm", "umm", "hum", "emmm", "err", "uh", "ah"
Filler word
or that they are about to talk): "like", "you know", mean", "okay", "so", "actually", "basically"
Hedging
Wordslexpressions used express ambiguity about what is saying (about to say or just said). Examples: "somewhat"; "I'm not an expert, but...", ", right?"; "... isn't it?": "I do not know...", "all know"; "I think..."
Correction
Corrects something they said. This focuses on the content of what is said, not on a syntax- -aadx error, or speech error.
Repetition
Emphasizing phrase by repeating, or repetition of a word, might be related to trying to find the right word or expression
Com-B Constructs
Positive
Statement positive feelings towards a behaviour or action.
Negative
Statement negative feelings towards behaviour or action.
Excuse
Statement where the participant mentions an excuse, a reason or justification for something that has happened hasn't happened. It can also be expres- sion of regret for doing/not doing something. Use of 'but', Shows avoidance lack of responsibility
Success
Statement of success with goal (focused the behaviour)
Fail
Statement of failure with goal (focused on the behaviour)
Cap
Mentions having the capability change their behaviour. Includes physical capability (e.g., balance, dexterity) psychological capability (e.g., knowl- edge, skills, memory).
No cap
Mentions NOT having capability to change their behaviour. Includes physical capability (e.g., balance, dexterity) or psychological capability (e.g., knowledge, skills, memory).
Mot
Mentions having motivation to change their behaviour. Includes reflective motivation (e.g., making plans, having positive attitudes/beliff)) or auto- matic motivation (e.g., desires, habit, feelings)
No mot
Mentions NOT having motivation to change their behaviour OR motivation NOT to change their behaviour. Includes reflective motivation (e.g., making plans, having positive attitudes/beliefs) or automatic motivation (e.g., de- sires, habit, feelings).
Opp
Mentions having opportunity change their behaviour. Includes physical opportunities (c.g.. access to financial resources, location, time) or social opportunity (c.g.. support/encourraeemmnt from others: norms)
No opp
Mentions NOT having opportunity to change their behaviour. Includes phys- ical opportunities (e.g., access to financial resources, location, time) or social opportunity (e.g., support/encouragemmnt from others; norms)
Table 11: BAH dataset annotation codebook: language cues.
38

---

Published as a conference paper at ICLR 2026
Audio cues
Definition
Pause
Briefly interrupting a sentence by having silent pauses in between words or ideas that differ from the usual pace of how the participants speaks. It includes silent pauses, paused speech or ideas.
Cut words
Ending speaking a word before completing the utterance of the word (e.g., say "exer... instead of "exercise"). Breaking the words or interrupting the words while they are being spoken. Might involve correcting syntax/speech
Slow
Reducing the speed of speaking. There is perceptible change in the speed while someone is talking, making it slower de-accelerated. It differs from paused speech or cutting off words since the words, phrases or ideas are not cut off or left the middle, there are no significant silences in the answers. can include elongating syllables words. Speed change is determined in comparison to the person's 's own baseline.
Fast
Changes in the speed of the answers, making it faster. Information given quickly. briskly lively. Speed change is determined in comparison to the person's 's own baseline.
Volume
Changes in volume of speech. Differences in how loud or quiet an answer shared, there can be differences in the volume of specific words or syllables. Includes: Raising volume, lowering volume, high volume, low volume, and mumbling. Vol- ume change is determined in comparison the person's own baseline.
Shaky voice
When there an rapid fluctuation or trembling of rhythm or tone (i.c., there is instability) to the way someone is speaking. It includes voice shaking, quivering in voice. Excludes case when shaking due to laughing
Breath
Audible breath, inhaling or exhaling, can be while the person talking or fore/after phrase. It includes changes in the breathing rhythm, intensity or deep- ness of the person (compared to the baseline) that create sound. Includes sigh, deep breath
Click
Quick sound made by pressing the tongue against the roof of the mouth or back of the teeth and snapping downward. It often signals disapproval, unsureness or impatience. The sound resembles "tsk" or "tsk-tsk."
Laugh
Engaging in laughter, variations thereof (e.g., snicker, chortle, giggle)
Stuttering
Involuntary repetition of sounds while speaking. This can be seen disruption or blocking of the speech by prolongation sounds or by struggling to say word or part of word. Even though the stutter might off word phrase it different since the person will finish the word or idea. Includes stammer, stumble.
Table 12: BAH dataset annotation codebook: audio cues.
39

---

Published as a conference paper at ICLR 2026
Body cues
Definition
Look away
Moving the orientation of the head away from the baseline position such that eyes or the gaze will look away. Includes the head facing down, head facing up, looking down, looking up, looking from side side, lowering head, raising head.
Shake
Turning the head from side side, or with a slight turn of the head to one or both sides. Includes shaking head "no". Rotation on the horizontal plane
be done with repetitive head movements
Tilt
Angling the head the side without focusing on something else, and holding the position. Changing the position of the head so it is in sloping position. It can be accompanied by changes the gaze but not necessarily. Includes head tilting up and down, tilting head to the side, tilted head. Includes bobbling head.
Throw
Throwing the head in rapid movement in particular direction.
Sigh
Movements of the chest, shoulder or head that accompany sigh or deep breath. includes long sigh, deep breath, sigh, big sigh. Noticeable bringing the chest or diaphragm muscles up and down. Change determined in comparison to the person's own baseline.
Nod
Moving the head up and down. Lowering and raising the head, can be done by slight or clearly marked movements. Includes movements such back and forward or a single small nod.
Shrug
Raising of the shoulders, it can be a momentary or slight rise or longer movement where one or both shoulders is raised. It includes shrugging shoulders, shrugs
Hands
Movements or placement of the hands that differs from baseline
Posture
Movements in the overall positioning of the spine, body or arms (independent from the head). The changes are determined by each person's 's baseline. It includes move- ments like readjusting in the seat, sloughing, turning to the sides. Needs to involve more than just the head. Excludes shrugging.
Scratch
Movements in the hands and arms to scratch or caress another part of the body or face. It includes scratching head, scratching neck, scratching eyes, scratching chin Rhythmic and repeated movements. Can be swaying, shaking, being jittery.
Restless
Table 13: BAH dataset annotation codebook: body cues.
Cross-modh inconsistency
Definition
FL
Face and language/speech do match. E.g., looking uncomfort- able while saying yes, looking annoyed or uncomfortable while saying they are happy, smiling while saying they are worried.
FA
Face and audio do not match. E.g., speaks in sad, energetic tone while smiling.
FB
Face and body do not match. E.g., Nodding while looking afraid or concerned, showing disgust but leaning forward
LA
Language/speech and audio do not match. E.g., speaks in sad, energetic tone while saying they are happy.
LB
Language/speech and body do not match. E.g., seems like they about to say something but do not, nod is discrepant with verbal speech, shaking head while saying yes Body and language/speech do not match. E.g., unengaged tone while nodding (in agreement)
AB
Table 14: BAH dataset annotation codebook: cross-modal inconsistency cues occurring simultane- ously.
40

---

Published as a conference paper at ICLR 2026
J BASELINE RESULTS
Visual features
AVGF1
Cropped faces
0.5028
0.0078
0.1923 + 0.0054
Full frame (body)
0.4781 + 0.0209 0.1774 0.0063
Head-pose
0.4321 0.0213 0.1915 0.0021
Table 15: Impact of visual modality type model performance test set of BAH frame-level classification. ResNet18 backbone is used. We report the average and standard deviation of repetitions with different seeds.
J.1 VISUAL MODALITY: CROPPED FACES VS FULL FRAME VS HEAD-POSE FEATURES
While cropped faces are the de facto visual modality used in affective computing, other visual fea- tures could be also relevant in A/H recognition. We explore here two additional visual features that are related to cues used by annotator to detect A/H. In particular, we consider the full frame which covers the body which is an important cue. In addition, covers the body cue "Look away". To estimate the head pose, we use the pretrained model To- kenHPE (Zhang al.) 2023a).. The backbone is frozen yield head-pose features, which are fol- lowed with an A/H classifier layer. Results obtained in Tab T5 show that while cropped faces and full frame yield competitive results, head-pose features have poor results in terms of AVGF This suggests that using only head-pore houd-poseTaa is not enough as many other cues are lost. However, it could be used model context. Additionally, cropped faces results are more stable. We note also that the full frame case contains noisy background which may have contributed in decreasing performance. Segmenting the body alone could yield better performance. all our experiments, we use cropped faces for the visual modality.
use head-pose estimation which
J.2 SUPERVISED LEARNING CASES (FOLLOWUP FROM MAIN PAPER)
Training details. For both cases, pretraining visual models RAF-DB (Li al. 2017), AffectNet (Mollahosseini et al. (2019), and Aff-wil ld2 (Kollias & Zafeiriou 2019), their fine- tuning, and final training with multimodal setup, we used learning rate between 0.0009 to 0.001 with multiplying coefficient of 10. When training on BAH, and in the case of using context, we used window size between 24 (1 second) to 2880 (2 minutes) with step of second (24 frames). In this case, we use a mini-batch size in (2, 4, 8, 16, 32), where sample in the mini-batch is window of frames; and a single-GPP training. In all trainings, we used weight decay of 0.0001. All our experiments were conducted on server with 4 NVIDIA A100 GPUs with 40 GB of memory, AMD EPYC 7413 24-Core Processor, and 503GB of RAM. We present in Table 16 the computation time of the multimodal case.
Ablation over the window length. We conduct an ablation to study the impact of the context (window length) on the performance of recognizing A/H. To this end, we use a window length from 24 to 3264 with a step of second (24 frames). Figure20jshows shows the obtained results. By considering WF1, performance improves with the increase of the context where it can reach above 0.825. On the other hand, AP prefers small context. Using a small context of few seconds could be a good compromise for all the metrics. Note that the average length of an A/H segment is around 4 seconds (96 frames).
Video-level classification. To obtain video level predictions, resort simple post-procs of frame-level predictions in the main paper. In particular, we follow (De-la Torre et al.) 2015). where sliding window averages the probability of the positive class each frame. The case where high probability in window suggests the presence of A/H in the video. In our case, we consider a context of second (24 frame). The maximum probability across all windows considered the video probability to have the positive class. The probability of the negative class is the complement of the probability of the positive one. We then proceed to measure the same performance metrics AVGF1, and AP. Similar to the main paper, we report the performance at video-level for different cases: using visual backbone only without and with context (Table 17). multimodal (Table 18).
41

---

Published as a conference paper at ICLR 2026
4ctetion. Window Length Chuncal va, AVG acore.
Aldeioon: Nirroww Langth (hremcal va, Agcroage Procason (AY).
M
Accragg Prectaion (401
0.39
0.39
M Mmmm
0.4
0.14
0.39
M
Langth
Langth
Figure 20: Impact of context (window) length the performance of frame-level classification when using visual modality alone (ResNet18): AVGF1, and AP. Best performance is indicated in red dot.
Case
Value
Train time epoch
5mins
Inference time per-frame
0.12ms
Total n. params.
223M
N. learnable params.
5M
N. FLOPs
1.87 TFLOPs
N. MACs
938 GMACs
Table 16:
Computation time, number of parameters, number of FLO TTOSs/MCCs for multimodal case with visual, audio and text (with ResNet for visual backbone). Visual backhone frozen, while audio and text backbones are used extract features offline and store them.
Without context
With context (TCN)
Backbone
AVGF1
AP
AVGF1
APVIT Xue ct al.
2022
0,5871
0.2882
0.6134
0.3703
ResNet18 (Hc
2016
0.4445
0.0993
0.4425
0.1437
ResNet34 Hc ct al.
2016
0.4419
0.1598
0.4448
0,2188
ResNe150 He et al.
2016
0.4405
0.1456
0.4442
0.1247
ResNet101 (He ct al.
2016
0.4236
0.1315
0.4448
0.1622
ResNet152 (He al.
2016
0.4715
0.1852
0.4316
0.2053
Video-FocalNet (tiny)
(Wasim et al. 2023
0.5294
0.6545
Video-FocalNet (small) Wasim al.
2023
0.5333
0.6558
Video-FocalNet (basc) Wasim et al. 2023)
0.5663
0.6732
Table 17: Visual modality performance on test set of BAH at video-level classification: impact of architecture and context.
Similar to frame-level results, using context and multimodal yields better performance. In addition, FAN fusion yields the highest performance over both metrics.
and fusion (Table 19
In addition, we conducted experiments on Video-FocalNer (Wasim et al. 2023) model, specialized in video-based classification. It can be trained to directly predict video class based on sampled frames. However, it accounts only for visual modality. We used cropped-and -nd--aan--aleddd faces as input. We experimented with different architectures: tiny, small, and base. Pretrained weights provided by (Wasim et al., 2023) over Kinetics-400 dataset (Kay et al., (017) are used for initialization. Fol- lowing (Wasim et al. (2023), we sampled 8 frames per-video. The stride between the frames depends on the video length. This is done during training and test. We used a batch size of 64 (videos) over GPUs. The model finetuned over BAH dataset for 60 epochs. Table Table shows that such specialized model yields better performance in average over both metrics. Although performance fluctuates be- tween the three architectures, base-case yields an AP of 0.6732 and AVGE1 of 0.5663, leading the board in average.
42

---

Published as a conference paper at ICLR 2026
Modalities
AVGF1
AP
Visual
0.4448
0.1622
Audio
0.4546
0,5306
Text
0,5589
0,4731
Visual + Audio
0,4428
0,1964
Visual + Text
0.5614
0.3407
Audio Text
0,4366
0.3148
Visual + Audio + Text
0,5934 0.3219
Table 18: Multimodal models performance on test set of BAH at video-level classification. For visual
modality, 52 backbone is used.
Fusion type
AVGF1
AP
LFAN
Zhang al
2023b
(0V3))
0.5934
0.3219
CAN
Zhang al.
2023b
(c1pru.2023)
0.4011
0.3206
MT
Walgorr
2024
(cvpru22024
0.4448
0.1069
JMT
Waligora
2024
(exprw.2024)
0.4448
0.0993
Table 19: Feature fusion performance on test set of BAH at video-level classification.
J.3 ZERO-SHOT INFERENCE: MULTIMODAL LARGE LANGUAGE MODELS (M-LLMS)
Multimodal LLMs (M-LLMs) have gained significant attention in the affective computing space due their ability to infer cross-modal dynamics across the visual. aural, and textual modalities. The problem of detecting ambivalence and hesitancy in videos is inherently multimodal as it requires also capturing the cross-modal inconsistency. To get out-of-the-box performance of existing SOTA M-LLM, performed zero-shot inference using the Vidco-LLaVA-7B-he (Lin et al. 2024). Since the performance of an M-LLM or LLMs in general can be heavily influenced by the query prompt, experiment with different variations of the prompts. Table 20 summarizes the different prompt variations used for zero-shot inference.
Prompt
Classify the emotion in the video as either 'Non-Ambivalent' or "Ambivalen'." Respond with only one word:
Simple
'Definition: Ambivalence is the state of having contradictory conflicting feelings or attitudes towards something or someone simultaneooslly. Classify the emotion in the video as either 'NonAmbiialenn' or A'miivalen'. Respond with only one word: 'Definition: Ambivalence and Hesitancy is understood as the simultaneous experience of desires for change and against change. Classify the emotion in the video as either Non-Ambiialent' or 'Ambivalent'. Respond with only one word:
Definition
Definition 2
'Video transcript: [transcript]. Definition: Ambivalence is the state of having contradictory or conflicting feelings or attitudes towards something or someone simultaneously. Classify the emotion in the video as either 'Non-Ambivalent' or 'Ambivalent.. Respond with only one word:
Transcript + Def
'Video transcript: (transcript). Definition: Ambivalence and Hesitancy understood as the simultaneous experience of desires for change and against change. Classify the emotion in the video either Moon-Amiialeen' or 'Ambivalent..
Transcript + Def 2
Respond with only one word:'
Table 20: Summary of prompt variations for zero-shot inference.
43

---

Published as a conference paper at ICLR 2026
J.3.1 FRAME-LEVEL PREDICTION
For frame-level prediction, adopt segment-wise strategy, where the entire video is divided into 8-frame chunks and passed through the model using a sliding window. This way, the model sees all the frames in each video. A single prediction obtained for the window, which is replicated for the segment to match the total number of frame labels in each video. The model's output, 'Non- Ambivalent or 'Amiivalent,, is mapped to and respectively match the ground truth.
Prompt
AVGF1
Simple
0.4416
Definition Only
0.4456
Definition Only
0.1651
Transcript Def
0.4535
Transcript Def
0.1849
Table 21: Frame Level Prediction using M-LLM.
Table 21 shows the results obtained for frame-level predictions using different prompts. The best results for frame-level prediction are obtained using the "Transcript + Def 1' prompt, where the actual transcript of the video is also provided, along with straightforward definition of A/H. The model performs better with more straightforward definition of the concept of ambivalence, with Definition
In the segment-wise approach applied with a sliding window of 8 frames, the model essentially sees every frame, but this approach limits the context window to be 1/3 of a second, which may not be enough to capture the temporal dependencies in the visual modality. We investigate the effeet of various lengths context windows on the overall performance.
Context Window
AVGF1
24 Frames
0.4539
48 Frames
0,4542
80 Frames
0,4535
120 Frames
0,4546
192 Frames
0,4540
Table 22: Performance comparison with increasing size context window for frame-level prediction.
We selected the best-performing query prompt from the first experiment (Table 21 to perform the ablation the context window size. Table 22 shows the results with different lengths of context window for the visual modality. 24 frames represent a one-second context window. Increasing the context window size 120 frames (5 seconds) only marginally improves the overall performance of the model, and it plateaus at 120 frames and then starts to drop which is an indicator that the visual encoder starts losing information with longer context window.
J.3.2 VIDEO-LEVEL PREDICTION
For video-level prediction, the entire video is fed to the model, and the transcript embedded the prompt. The model selects 8 uniformly spaced frames from the video and predicts single output. Similar to frame-level predictions, the model's output is mapped to and 1, and the performance metrics calculated.
Table presents the video-level prediction results. Similar to frame-level predictions, the 'simple' prompt without any context the definition or the transcript performs the worst and predicts all samples to be Aobiivllvaln similar trend also observed here, ie., adding the definition and the transcript substantially affects the model performance.
44

---

Published as a conference paper at ICLR 2026
Prompt
AVGF1
Simple
0,2827
Definition Only
0.3326
Definition Only
0.3772
Transcript Def
0.6341
Transcript 2
0.3945
Table 23: Video Level Prediction using M-LLM.
J.3.3 ANALYSIS AND DISCUSSION
The performance of M-LLM with zero-shot inference is substantially influenced by the query prompt. As observed from tables 21 and 23 simply asking the model to predict emotion based on the visual modality only performs the worst, whereas adding only the definition of A/H in the query prompt helps the model better identify the positive(A/H) class. Best results in all cases are obtained with the introduction of the text transcript of the video in the query prompt. We conjecture that this happens for two reasons: i) the textual modality serves a significant role in the identification of the A/H class, and ii) the current M-LLMS' performance heavily reliant on the textual modality. This coheres with the overall structure of traditional M-LLMs that are built upon well-trained LLMs with the addition of a visual encoder like ViT, which used encode the visual information that is fed to the LLM for downstream tasks. Intuitively, the performance should increase with careful fine-tuning on the BAH dataset.
Further, the idea of textualizing the aural and visual modalities explored in (Richet et al. 2024 can be well-suited for a task like this where the audio and visual modalities essentially summarize the cues detected in the corresponding modalities. Particularly for tasks like subtle emotion recognition or the detection A/H, where cross-modal inconsistency has to be considered. Textualizing the aural and visual modalities can be done adequately exploit the reasoning abilities of SOTA LLMs.
J.4 PERSONALIZATION USING DOMAIN ADAPTATION
Domain adaptation (DA) (Han et 2020 Li Deng 2018) has emerged promising approach
for personalized expression recognition, where the model is trained on diverse labeled source data to generalize to unlabeled target domains representing individual users. Recent research emphasizes
on subject-based domain adaptation (Sharali et al. 2026 2025| Zeeshan et al. 2024 2025), where each individual is defined as distinct domain. DA will be employed to personalize ML models by considering each participant in the test set separate target domain.
Experimental Protocol. For personalized in BAH, we adopt the standard protocol from prior work (Zeshan al. 2024 Sharafi al. 2025). which involves partitioning the data of each target indi- vidual into train, val validation, and test sets. Given the class imbalance the BAH dataset, we ensure a balanced representation of positive and negative samples within each split. We establish the follow- ing baseline methods to evaluate the effectiveness of personalized BAH recognition: Source-only: The model trained on the source data is directly evaluated on the target individual test set without any adaptation. This assesses the generalization capability of the source model. Unsupervised Domain Adaptation (UDA): Source data with labels utilized adapt the model each target individual using unlabeled data. This explores the potential of leveraging source knowledge for personalization in the absence of target labels. Source Free Unsupervised Domain Adaptation (SFUDA): Adap- tation is performed solely using the unlabeled data from the target individual, without access to the source data. This examines the feasibility of personalization when the source data is unavailable. Oracle: The model is fine-tuned using the labeled data from the target individual during training. This provides an upper-bound bound performance, representing a fully supervised model.
Visual backbone. We employ ViT based model for personalization, leveraging its superior perfor- mance over BosNet-based architectures for visual tasks without contextual information. all our experiments, utilize ViT-based model pre-trained on the source data.
45

---

Published as a conference paper at ICLR 2026
Methods
AVGE1
Source-only
0,4894
0.0999
0.3565 0.1844
UDA (MMD) Sejdinovic al.
2013
0.4931
0.0943
0.3589 0.1831
UDA (Sub- Bused)
(Zeeshan el m
2024
(fe 2024)
0.5417
0.0728
0.3739
0.1789
SFUDA (SHOT)
Liang ct al.
2020
2020)
0.4919 1056
0.3520 = 0.1666
SFUDA (NRC)
Yang al.
222
(neurips.2021)
0.5174 1: 1.1041 0.1041
0.3688 0.1487
Orucle
0,3864 0.0751
0.4181 + 0.1750
Table 24: Performance of UDA and SFUDA with Source-only and Oracle on BAH. Results are reported the average over all target subjects (mean + standard deviation).
Over years
(a) Comparison between different age groups on DA (b) Comparison between different DA methods based on participant Sex.
methods.
Figure 21: Comparison between different age groups and DA methods.
J.4.1 UNSUPERVISED DOMAIN ADAPTATION:
We investigated two unsupervised domain adaptation (UDA) approaches for personalized BAH recognition: (i) a discrepancy- -hasedd method using Maximum Mean Discrepancy (MMD) (Sejdi- novic al. 2013) to minimize the domain gap and improve performance the target subject, and (ii) a subject-based (Zeeshan et al.) 2024) method using self-supervision that trains the model by generating pseudo-labels for the target domain, followed by reducing the domain shift using MMD that aligns source and target.
Implementation detail. In UDA experiments, we optimize our model using Stochastic Gradient Descent (SGD) (Sutskever et al.) 2013) with a learning rate of 2e - 4, momentum of 0.9, weight decay of 5e 4, and cosine annealing scheduler (Loshchilov & Hutter (017)) with a minimum learning rate of 2e 5. We set the batch size to 64 and run each target adaptation for 10 epochs. For the subject-based method, we introduce a hyperparameter % 0.01 weight the target loss, computed using pseudo-labels generated by the Augmented Confident Pseudo-Label (ACPL) tech- nique (Zeeshan et al. 2024). This weighting is essential for mitigating noise in the pseudo-labels, in conjunction with a confidence threshold of 0.95 that is updated every 4 epochs.
J.4.2 SOURCE FREE UNSUPERVISED DOMAIN ADAPTATION
Two source-free unsupervised domain adaptation (SFUDA) approaches were explored for personal- ized BAH recognition: (i) representation learning strategy inspired by hypothesis transfer (Liang al.] 2020). where information maximization was used to adapt the model to the target do- main, and target-specific prototypes guided pseudo-labelling for class-level alignment, and (ii) neighbourhood-basee method (Yang et al., 2021) in which label consistency was encouraged among target features and their reciprocal nearest neighbours, while expanded neighbourhoods were used to aggregate local structure and reduce the impact of noisy supervision through self-regularization and weeintt-wwightedd loss.
We optimize the model using SGD with a learning rate of 1e - 4, momen- tum of 0.9, and weight decay of le 3. The model is trained for 30 epochs with batch size of 64. For NRC-based adaptation, we maintain memory banks of target features and predictions to retrieve K nearest neighbours and M expanded neighbors.
Experimental protocol.
46

---

Published as a conference paper at ICLR 2026
J.4.3 RESULT AND ANALYSIS
The average performance acToss target participants, along with the standard deviation, is presented in Table 24 All reported results are based on evaluations of the respective target test sets. Our analysis demonstrates the effectiveness of domain adaptation for personalized detection of the A/H class in the BAH dataset. All tested methods surpass the Source-only baseline in AVGF1 and AP positive-class metrics. Notably, Sub-based achieves the highest AVGF1 (0.5417) and AP (0.3739), outperforming other domain adaptation techniques. While MMD and SHOT yield only modest gains, they still lag behind the Sub-based method, underscoring the advantage of pseudo-labeling for boosting minority-class recall and precision. In contrast, NRC achieves a more substantial im- provement of around 3 percentage points over Source-only, indicating that even with some sensitiv- ity to domain shifts, it more effectively exploits target-domain structure. The Oracle upper bound (AVGF1: 0.5864) underscores the substantial potential for further advancements in positive- detection within this context. Even slight degradations in negative-class performance disproportion- ately impact the overall AP score. For example, Sub-based method emphasizes enhancing positive class identification, likely incurs a cost in precision or recall on the more frequent negative class, necessary compromise to effectively detect the A/H class.
Age-wise analysis. Figure 21a illustrates the varying impact of age on DA methods. Overall per- formance tends to peak in the 35-44 years group across most methods, while the 25-34 years group generally shows the lowest AVCFL. Sub-based consistently achieves the highest or near-highest performance in all age groups, with clear gains over Source-only, and is only slightly outperformed by SHOT in the Over 65 yeurs group. SHOT remains competitive across all ages and, together with Sub-based, yields the largest improvements for the youngest (25-34 years) and oldest (Over 65 years) subjects. In contrast, MMD and NRC provide only modest gains over Source-only in the 25-34 years and 35-44 years groups and can even underperform Source-only for 45-54 years, high- lighting their sensitivity to age-related domain shifts. Finally, Source-only has consistently lower Avg Fl across all age groups, confirming the benefit of DA, particularly for the more challenging age ranges.
observe that the female category generally exhibits higher values across most methods compared to the male. Specifically, female shows the highest values in NRC (0.0.52), SHOT (0.0.49), and MMD (0.50). It can also be noted that the number of female subjects is equal to the male subjects.
Sex-wise analysis. In the Figure 21b
47


Reviews:
[{"review_id": "kG5zfUmzDb", "reviewer": "Reviewer_tpao", "summary": "The paper proposes BAH, a dataset for recognizing ambivalence and hesitancy (A/H) in short webcam-style videos. The dataset consists of 1118 videos (~8 h total) from 224 participants across 9 Canadian provinces, each responding to 7 prompts. The dataset includes both frame-level and video-level A/H annotations. Baseline models are provided for visual, audio, and text modalities, as well as simple fusion and contextual vs non-contextual comparisons.", "questions": "1. The contextual and multimodal (especially tri-modal) results are very close to those of single-modality models. How do you explain this? Why did tri-modal fusion underperform pairwise combinations? Are these differences statistically significant across runs? \n2. Given that ambivalence is highly context-dependent, what temporal window or duration do you consider sufficient for a valid A/H judgment? How sensitive are the annotations or models to this choice? \n3. How was ground truth established when annotators disagreed? Was it based on majority vote, adjudication by a lead annotator, or consensus discussion? \n4. The codebook explicitly includes body language cues, yet the modeling pipeline omits body or pose features. Why didn’t you include these, and do you plan to in future work? \n5. Can you provide inter-annotator reliability metrics (e.g. Cohen’s κ) for A/H labels? Also, how were temporal boundaries defined and aligned between annotators?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The first publicly available dataset specifically focused on ambivalence/hesitancy detection.\n2. Clearly described data collection and annotation framework, incorporating both global- and frame-level labels", "weaknesses": "1. All subjects are from a single country (Canada), which may limit cultural and linguistic generalization of ambivalence expressions.\n2. Contextual and multimodal (tri-modal) results are unexpectedly similar to non-contextual or single-modality baselines, suggesting limited exploitation of temporal or cross-modal dependencies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EnZhn2l9rY", "reviewer": "Reviewer_RBNh", "summary": "This paper introduces the first Behavioural Ambivalence//Hesitancy (BAH) dataset collected for subject-based multimodal recognition of A/H in videos. It contains videos from 224 participants captured across nine provinces in Canada, with different age, and ethnicity. BAH contains 1,118 videos for a total duration of 8.26 hours with 1.5 hours of A/H. The paper also provides preliminary benchmarking results using baseline models trained on BAH for frame- and video-level recognition with mono- and multi-modal setups. It also includes results on models for zero-shot prediction, and for personalization using unsupervised domain adaptation. The limited performance of baseline models highlights the challenges of recognizing A/H in real-world videos.", "questions": "1. In table 5, it is shown that with Visual + Audio + Text, the results are worse than other cases, can the authors explain the possible reason? \n\n2. What are main challenges of ambivalence and hesitancy (A/H)  recognition compared with other types of emotion recognition such as anger or happy?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. It is interesting to have a dataset for ambivalence and hesitancy (A/H)  recognition which  involve subtle and conflicting emotions that are manifested by a discord between multiple modalities, such as facial and vocal expressions, and body language. \n\n2. The authors conducted extensive experiments to demonstrate the potential of this dataset for frame- and video-based emotion recognition.", "weaknesses": "1. In the experimental results, the authors only consider CNN- and ViT-based models which are originally designed for image classification. It would be more interesting to consider video-based models to demonstrate the difficulty of the proposed benchmark.\n\n2. While a dedicated dataset for ambivalence and hesitancy (A/H)  recognition is interesting, it is not clear what is the major difference between this task compared with other video recognition task such as activity recognition. In other words, what aspect makes this task challenging or difficult. \n\n3. The dataset size is relatively small with only a total duration of 8.26 hours which can possibly limit the usefulness of the proposed dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YEIR415W4Z", "reviewer": "Reviewer_GdzA", "summary": "This paper presents the Behavioural Ambivalence/Hesitancy (BAH) dataset, a multimodal video corpus for recognizing ambivalence/hesitancy (A/H) states. It includes 1,118 videos from 224 participants across Canada, annotated at video- and frame-levels with onset–offset segments and multimodal cues (face, audio, text). The dataset aims to model complex, sustained emotional states rather than discrete peaks. Baseline experiments perform binary A/H vs. non-A/H classification, showing the task’s difficulty and establishing a benchmark for future affective computing research.", "questions": "1. As mentioned in weaknesses. We suggest distinct recognition of ambivalence and\nhesitancy and a connection to behavioural change, while the paper only conducts a binary A/H vs. non-A/H classification without behavioural intervention. A clearer alignment between the task scope and the title would improve conceptual precision.\n2. The abstract could be more concise and focused on the dataset’s conceptual contribution and key findings rather than listing detailed statistics. Streamlining this section would make the paper’s main message clearer.\n3. In the Introduction, the motivation for detecting A/H remains vague. The paper would benefit from a clearer explanation of why distinguishing A/H from non-A/H emotions is important and in which practical contexts this task has value. At present, the classification objective seems somewhat detached from real-world applications. The connection to behavioural change also remains conceptual. Clarifying how A/H recognition could be used in adaptive feedback or intervention systems would make this link more convincing.\n4. Finally, the explanation of the multimodal fusion results (Table 5 / Table 15) is not entirely satisfactory. The paper attributes the drop in performance with tri-modal fusion (visual + audio + text) to “modality conflicts.” However, given that ambivalence and hesitancy are themselves conflictive emotional states, such cross- modal inconsistency might in fact be an informative signal rather than noise. It is unclear why three-modality fusion causes conflict while two-modality setups do not, especially since the text modality is derived from speech. Further clarification and analysis of this phenomenon would strengthen the interpretation of the results.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces a novel multimodal dataset that integrates visual, audio, and textual modalities, focusing on the recognition of ambivalence and hesitancy (A/H), which are complex, sustained emotional states rather than discrete basic emotions. It thereby proposes a new affect recognition task that broadens the traditional emotion recognition paradigm.\n2. The dataset is of moderate scale, containing videos from 224 participants across 9 Canadian provinces, ensuring demographic and ethnic diversity and supporting fairer modelling of human affect. This makes it a valuable contribution to the affective computing community.\n3. The baseline experiments are well designed and relatively comprehensive, covering frame-level and video-level recognition in both unimodal and multimodal settings. The paper also reports zero-shot prediction results and unsupervised domain adaptation models for personalization, which further enhance the dataset’s utility and research relevance.", "weaknesses": "1. While the dataset itself is valuable, the title and task definition are misleading. The term “Ambivalence/Hesitancy Recognition” suggests separate classification of ambivalence and hesitancy, yet all experiments address a binary detection task (A/H vs. non-A/H) without distinguishing the two.\n2. The inclusion of “for Behavioural Change” in the title appears overstated, as the study does not involve any behavioural intervention, longitudinal tracking, or pre– post change analysis. The link between recognizing A/H emotions and actual behavioural change is discussed only conceptually, not empirically demonstrated.\n3. The scientific relevance of detecting A/H remains insufficiently justified. While ambivalence and hesitancy are theoretically relevant to behaviour regulation, the paper does not clarify how A/H recognition could inform or improve behavioural interventions. Strengthening this conceptual bridge would significantly enhance the contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eOtk0nsOIz", "reviewer": "Reviewer_pr3G", "summary": "The paper introduces a new multimodal dataset for recognizing ambivalence and hesitancy. It includes recordings with expert annotations covering facial, vocal, verbal, and bodily cues, offering a valuable resource for studying complex emotional states relevant to behavior change. The authors provide baseline experiments across different modalities and fusion strategies, showing that temporal context and multimodal learning improve performance but that the task remains challenging, paving the way for future research on nuanced emotion understanding.", "questions": "1. What are the inter-annotator agreement (IAA) scores at both the video and frame levels for Ambivalence/Hesitancy labeling, as well as for cue tagging? Additionally, how do the models perform on high-certainty versus low-certainty segments?\n2. How did the annotators distinguish ambivalence from mixed emotions or uncertainty without intent conflict? Please provide annotated examples and an error typology to clarify how these cases were handled.\n3. Please include metadata statistics for the training, validation, and test splits, especially regarding demographic distribution, to assess data balance and fairness.\n4. It would strengthen the paper to clearly explain the conceptual and methodological differences between Ambivalence/Hesitancy recognition and temporal emotion recognition, highlighting why A/H detection requires distinct modeling strategies.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The first public dataset focused on recognizing ambivalence and hesitancy, emotional states that play an important role in behavior-change research but have received little attention in machine learning.\n2.The authors conducted comprehensive experiments comparing different setups: single-modality vs. multi-modality, with or without temporal context, and various fusion methods. The results show that adding temporal context improves performance, simple concatenation often works surprisingly well, and combining all three modalities does not always outperform simpler setups, indicating that multimodal fusion for ambivalence recognition remains a challenging problem.", "weaknesses": "1. The dataset provides only binary (0/1) labels for Ambivalence/Hesitancy, which may be too simplistic. Incorporating continuous annotations such as valence–arousal or PAD scales could enable a more fine-grained analysis of emotional states and cues.\n2. Although the dataset includes facial, linguistic, audio, and body cues, these are annotated only at the video level. While this adds interpretability, it limits temporal precision. Adding timestamps or ideally frame-level cue annotations would make the dataset far more useful for detailed temporal modeling.\n3. The paper describes the annotation process and co-annotation procedures but does not report quantitative inter-annotator agreement. Without this metric, it is difficult to assess the reliability and consistency of the labels.\n4. The paper does not provide metadata statistics for the training, validation, and test splits, particularly regarding demographic balance. Reporting and ensuring demographic diversity across splits would support fairness analysis and enhance the dataset’s research value.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)