StressID: a Multimodal Dataset for Stress Identification
Authors: Hava Chaptoukaev
Valeriya Strizhkova
Michele Panariello
Bianca Dalpaos
Aglind Reka
Valeria Manera
Susanne Thummler
Esma ISMAILOVA
Nicholas Evans
Francois Bremond
Massimiliano Todisco
Maria A Zuluaga
Laura M. Ferrari
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Stress identification
dataset
TL;DR: StressID is the a novel and large dataset for stress identification with three different sources of data and three classes of stimuli.

Abstract: StressID is a new dataset specifically designed for stress identification from
unimodal and multimodal data. It contains videos of facial expressions, audio
recordings, and physiological signals. The video and audio recordings are acquired
using an RGB camera with an integrated microphone. The physiological data
is composed of electrocardiography (ECG), electrodermal activity (EDA), and
respiration signals that are recorded and monitored using a wearable device. This
experimental setup ensures a synchronized and high-quality multimodal data col-
lection. Different stress-inducing stimuli, such as emotional video clips, cognitive
tasks including mathematical or comprehension exercises, and public speaking
scenarios, are designed to trigger a diverse range of emotional responses. The
final dataset consists of recordings from 65 participants who performed 11 tasks,
as well as their ratings of perceived relaxation, stress, arousal, and valence levels.
StressID is one of the largest datasets for stress identification that features three
different sources of data and varied classes of stimuli, representing more than
39 hours of annotated data in total. StressID offers baseline models for stress
classification including a cleaning, feature extraction, and classification phase for
each modality. Additionally, we provide multimodal predictive models combining
video, audio, and physiological inputs. The data and the code for the baselines are
available at https://project.inria.fr/stressid/.

StressID: a Multimodal Dataset for Stress
Identification
Hava Chaptoukaey'
Valeriya Strizhkova
Michele Panariello'
Bianca D'Alpaos'
Aglind Reka?
Valeria Manera Susanne Thümmler
Esma Ismailova
Nicholas Evans?
François Bremond 2.3
Massimiliano Todisco! Maria A. Zuluaga Laura M. Ferrari2.5
EURECOM, Sophia Antipolis, France
Inria, Valbonne, France
CoBTek, Université Cote d' Azur, Nice, France
Mines Saint-Erienne, Centre CMP, Gardanne, France
The Biorobotics Institute, Scuola Superiore Sant' Anna, Pontedera, Italy
{chaptouk, panariel, panaaiel evans, todisco, zuluaaaa/euurecom fr [valeriya strizhkwaaaaaaaaaaaaaaaaoo francois. bremond, www.a...raarraaa.......aa ferrari)@inria.....
Abstract
StressID is new dataset specifically designed for stress identification from unimodal and multimodal data. It contains videos of facial expressions, audio recordings. and physiological signals. The video and audio recordings acquired using an RGB camera with an integrated microphone. The physiological data is composed of electrocardiography (ECG), electrodermal activity (EDA), and respiration signals that are recorded and monitored using a wearable device. This experimental setup ensures synchronized and high-quality multimodal data col- lection. Different stress-inducing stimuli, such as emotional video clips, cognitive tasks including mathematical or comprehension exercises, and public speaking scenarios, are designed to trigger a diverse range of emotional responses. The final dataset consists of recordings from participants who performed tasks, well as their ratings of perceived relaxation, stress, arousal, and valence levels. Stress ID one of the largest datasets for stress identification that features three different sources of data and varied classes of stimuli, representing more than 39 hours of annotated data total. StressID offers baseline models for stress classification including cleaning, feature extraction, and classification phase for each modality. Additionally, we provide multimodal predictive models combining video, audio, and physiological inputs. The data and the code for the baselines are inria. fr www.srrrsstsit stressid/
available at
Introduction
While a healthy amount of stress is necessary for functioning daily life, it can rapidly begin to negatively impact health and productivity when it exceeds an individual's coping level. Negative stress can be triggering or aggravating factor for many diseases and pathological conditions [16], and frequent and intense exposures to stress can cause structural changes in the brain with long- term effects on the nervous system [10] Monitoring of stress levels could play a major role in the prevention of stress-related issues, and early stress detection is vital in patients exhibiting emotional disorders, or high-risk jobs such as surgeons, pilots long- aingdiitance drivers.
Equua contribution.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Trams
Seep VN
camera
Time
Treb
Figure I: Data collection set-up of StressID.
In the last few years, machine and deep learning have been playing major roles in stress recognition research. In practice however, building robust and reliable models for stress identification requires; 1) understanding and integrating the differences between subgroups of the population ensure bias free applications. 2) integrating the relationships between physical and physiological responses to stress, 3) studying responses various categories of stressors, the perception of stress differs strongly from one individual to another. An essential element to such analyses is high-quality and versatile multimodal datasets that include varied categories of stressors, and recorded large and diverse populations. However, existing datasets do not answer these needs. They are generally restricted in size (i.e. a few dozen participants) and majority is focused on a single source of data (i.e. physiological signals, video or audio) although multimodal datasets have considerable advantages 226 29 Moreover, existing datasets often provide imbalanced subject responses, due to both an inability of the recording protocol to elicit strong reactions and a lack of diversity in the stimuli making it difficult to deploy deriving analyses to real-life applications.
To address these limitations, we propose StressID, a novel multimodal dataset with facial video, audio, and physiological data. To the best of our knowledge, StressID is one of the largest available multimodal dataset in the field that includes varied stimuli. It composed of 65 subjects and more than 39 hours of annotated data in total. Stress is designed specifically for the identification of stress from different triggers, by using guided breathing task, 2 video clips, 7 different interactive stressors, and relaxation task. As illustrated in Figure StressID uses a collection of wearable sensors to record the physiological responses of the participants, namely, an Electrocardiogram (ECG), an Electrodermal Activity (EDA) sensor, and respiration sensor. The data coupled with synchronized facial video and audio recordings. Each task is associated with 6 different annotations: 4 scores from a self-assee assessment rating perceived stress and relaxation, along with valence and arousal based on the Self- Assesmment Manikin (SAM) [9: and 2 discrete labels derived from the 4 seff-assessments. These data annotations serve to train supervised models.
We summarize our main contributions as follows:
A novel multimodal dataset focused on stress-inducing tasks, composed of ECG, EDA, respiration, facial video, and audio recordings. The modalities are synchronized and anno- tated with self-assessments from the participants evaluating their levels of relaxation, stress, valence, and arousal.
An easy to reproduce experimental protocol for recording behavioral and physiological responses diverse triggers, using wearable and global sensors.
Instructions for using the presented dataset and an open-source implementation of sev- eral baseline models for stress recognition from video, audio, and physiological signals respectively, well as multimodal models combining the three inputs.
The remainder of this paper is organized as follows. Section 2 provides an overview of the existing datasets for stress recognition. Section Jdetiribesserbbess the dataset design and its contents. In Sections we present multiple baselines for stress detection using machine learning. In Section 5|and 6 we discuss the limitations and ethical considerations of our work. Finally, we summarize our work and discuss future directions.

---

2
Related work
Table Ppaces StressID in the context of related stress recognition datasets. The SUS datasets [52] gather the recordings of 35 subjects collected during aircraft communication training. This unimodal collection of datasets only features audio recordings without self-assessments or external annotation and employs an uncommon elicitation task. SADVAW [55] is a dataset composed of 1236 video clips from 41 Korean movies, making the setting closer to the real world and including broader range of responses. However, it features video recordings exclusively, restricting deriving applications to computer vision systems only. Among the works investigating the physiological aspect of stress, collects physiological data from 9 subjects exposed to driving-relatee
tasks. The lack of self-assessment or external annotations significantly limits the accuracy of measuring stress. In addition, the dataset is collected in the very specific setting of driving, with narrow range of stressors considerably restricting its usage. WeSAD 48 and CLAS [36], two of the most widely explored datasets for stress recognition, contain physiological data from 15 and 62 subjects respectively. collected using wearable devices. The participants partake in various tasks, combining perceptive stressors in the form of audiovisual stimuli, with several variations of the Trier Social Stress Test (TSST) However, they do not include any behavioral modalities.
DriveDB 25
There exist a few multimodal datasets for stress recognition, such as MuSE 27 and SWELL- KW 33. They feature broader set of modalities and are collected in laboratory environments imitating real-life activities. MuSE participants are elicited through audiovisual and public speaking tasks. SWELL-KW participants perform office work several topics designed elicit different emotions. These datasets are limited in size with recordings of respectively 28 and 25 subjects. Finally, the distracted driving dataset (54) gathers recordings of 68 subjects in the setting of simulated driving with stress-inducing distractions. The lack of diversity in the stimuli restricts subsequent applications the setting of driving. Moreover, cardiac activity is acquired in terms of heart rate, which does not allow the extraction of heart rate variability (HRV) measures, key measure in the identification of stress 30)
Comparison with the state-of-the-art. StressID aims to fill the gap in the existing related work. It features both physiological and behavioral modalities, includes a large number of participants, exploits varied stimuli, and includes participants replies to 4 self-assessment questions providing insights on the subject's emotional state. Although CLAS (36] and WeSAD 488 present similar experimental set-ups, they focus on physiological modalities and do not include behavioral data.
Instead, StressID features three types of modalities: video, audio, and physiological signals capturing complementary information. While MUSE (27) and SWELL-KW [33] are also multimodal
datasets recorded in similar conditions, they are very limited in size. On the contrary, with 65 subjects recorded StressID is one of the largest datasets designed for stress identification. Finally, although the size and modalities of the distracted driving dataset [54] are comparable to StressID, it relies on very anvironment-specife stressors, whereas StressID includes emotional video-clips, cognitive tasks, and social stressors based on public speaking. which represents a key aspect to guarantee the collection of a wide range of responses. To summarize, Itress is the first multimodal dataset for stress identification that is recorded on a large number of participants but also features a wide range of stimuli ensuring more versatility in deriving applications.
Table 1: Comparison of StressID to related datasets.
Dataset
#Subjects
Modalities
Stressors
Data annotations
SUS
Speech
Airezaft communication training
Stressor based
SADVAW
Video
External annotations
DriveD8
HMG, BDA, BCG, HR. Respiration BCG, EDA, EMG, BVP. Respiration, Temperature, Acceleration
Driving tasks
Stressor-based
Stressor- PANAS 56
WoSAD
15
TSST. Audiovisual
STAI
Cognitive load, Audiovisual
CLAS
62
BOG, PPG, EDA, Acceleration
SAM
MuSE
HDA, HR, Breath rate, Temperature, Face and upper body video, Audio
Public speaking. Audiovisual
PSS 14
SAM, External
amnotations
ECG, EDA. Face and upper body video, Posture, Computer logging
Office work with interruptions and pressure
NASA task load
24
SAM.
SWELL-KW
25
Stress
Distracted Driving
68
EDA, HR sigual, Respiration, Face video,
Simulated driving with distractions
Shessor-bsed NASA task load, SAM
dataset
Driving performances
EDA, ECG, Respiration, Face video,
Cognitive lond, Public speaking, Audiovisual
StreseID
Speech
SAM, Stress assessment
3

---

BLOCK VIDEO CLIPS
BREATHING
7 INTERACTIVE STRESSORS
RELAX
BACKWARD from suberecting for as you can)
SELF ASSESSMENT
SELF ASSESSMENT
SELF ASSESSMENT
SELF ASSESSMENT
NEUTRAL BASELINE
MOST STRESSFUL
Figure 2: Overview of the experimental protocol. The experiment consists of 11 tasks divided into four blocks: guided breathing task, emotional video clips, 7 interactive stressors, and relaxation task.
3 StressID Dataset
StressID takes a step towards building more robust and reliable applications for automated stress identification by enabling the design of versatile and bias-free algorithms. To achieve this, Stress features responses to several categories of stress-inducing stimuli to account for the variability of responses from one individual to another, rather than focusing on single task. In addition, the large number of participants of StressID enables analyses of the demographics associated with stress, based on factors such as gender or age, thus advancing towards reducing representation bias by integrating these differences in subsequent algorithms. Ultimately. the design of the StressID dataset supports a variety of learning pipelines, by offering possibilities for the analysis of subject- specific, task-specific and modality-specific responses to stress. We describe the design of StressID Section 3.1 We then introduce the resulting dataset in Section 3.2, and outline our data annotation process in Section
3.3
3.1
Dataset Design
3.1.1 Experimental Protocol
Figure 2 illustrates the experimental protocol used to collect StressID. It consists of 11 tasks separated by self-assessments and grouped into 4 blocks: guided breathing, watching emotional video clips, a sequence of interactive tasks, and a relaxation phase. Tasks have been designed to elicit different categories of responses; 1) stimulate the audiovisual cortex of the participants, 2) increase the cognitive load by soliciting attention, comprehension, mental arithmetic or multi-tasking abilities, and 3) elicit psycho-social stress leveraging on public speaking as stressor. All stimuli easy to implement and do not require any special setup [5) The full instructions given to participants are provided in Appendix D.2.
The first block of the protocol consists of the single task of Breathing. The participants watch a guided breathing video of 3 minutes. It aims to relax and reset to neutral the emotional state of the subjects. This recording is used as baseline for nen-verbal neutral state of each participant. After the breathing task, the participants count forward for minute.
Guided breathing.
Emotional video-clips. This block consists in watching emotional videos clips, retrieved from the FilmStim database [47] These videos have been selected to elicit specific emotional responses.
Video! an extract from the movie There's 's something about Mary, selected to elicit low arousal and positive valence in the participants.
Video2 an extract from the movie Indiana Jones and the Last Crusade, selected to elicit high arousal and negative valence.
Interactive tasks. This block consists of sequence of 7 interactive stressors based on well- established clinical methods to induce stress [7]. All the tasks have striet requirement for response in minute and the order of the stressors is designed to be unexpected to the participants.

---

Countingl: Mental Arithmetic Task (MAT) designed to increase the participants' cognitive load through arithmetic operations with varying range of difficulty. In this task, the participants receive the instructions to count backwards from 100 subtracting 3 as fast as they can.
Counting2: : another MAT of increased difficulty. Participants are asked to count backward from 1011 subtracting 7 as fast as they can.
Stroop a variant of the Stroop Color-Wore Test (53), selected to increase the cognitive load by soliciting the attention and reactivity of the participants.
Speaking: Social Evaluative Task (SET), leveraging public speaking as social stressor. The subjects are instructed to explain their strengths and weaknesses, emulating stressful job interview conditions.
Math task designed increase the mental workload. The participants asked to resolve 20 mathematical problems in one minute.
Reading a task composed of 2 phases and designed as a TSST variation. Participants have read text, in the first step, and then explain what they read, in the step, thus simultaneously soliciting comprehension abilities and using speaking as a stressor.
Counting3: MAT with added difficulty. Participants are instructed to count backwards from 1152 subtracting 3, as fast as they can, while repeating an independent hand movement. This task is designed to increase the mental workload by soliciting participants multi-tasking abilities.
At the end of the third block, the participants asked to designate the task perceived most stressful.
Relaxation. The last block of the experimental protocol is solely composed of the Relax task. It consists of minute and 30 seconds long relaxation part, where participants are instructed to watch relaxing video 233
3.1.2 Sensors
Three different physiological signals are collected in StressID: electrocardiogram (ECG), electro- dermal activity (EDA), and respiration signal. They are recorded using the BioSignalsPlux acquisition system The BioSignalPlux kit consists of 4-channel hub communicating via Bluetooth with the OpenSignals (R)evolution platform for data visualization and acquisition, connected to an ECG, EDA, and respiration sensor. The hub ensures the synchronized recording of up 4 sensors simultaneously. The ECG is acquired with 3 Ag/AgCI electrodes located on the ribs of the non- dominant side of the subjects. The EDA is measured with 2 Ag/AgCI electrodes attached to the palm of the non-dominant hand. The respiration is measured through a chest belt with an integrated piezoelectric sensing element. The selected devices have high signal-to-noise ratio all physiological signals are acquired with sampling rate of 500 Hz and resolution of 16 bits per sample.
42 4 44 and
The video and audio are acquired using a Logitech QuickCam Pro 9000 RGB camera with an integrated microphone. The video is acquired with 720p resolution and a rate of 15 frames per second. The audio is recorded at a sampling rate of 32kHz and a resolution of 16 bits per sample.
3.2 Dataset Description
3.2.1
Recruitment and Recording
Most of the participants of StressID are Science, Technology, Engineering, and Mathematics (STEM) students and workers. In total, 65 healthy participants without compensation. They included 18 women and 47 men of ages ranging between 21 and 55 years old (29y.0. + 7). Among the participants, 32% were master students and interns, 20% PhD students, and the remaining 48% represented diverse tertiary professions. All subjects were required to have sufficient proficiency in English and they were requested to sign consent form participate.
recruited voluntary basis,
biosignalsplux, PLUX wireless biosignals S.A. (Lisbon. Portugal)
5

---

The participants could either consent to, Option A: research use and public release of all their recorded data, including identifying data (i.e. physiological, audio, and video). Option B: research use of all their recorded data, but no public release of identifying data (i.e. only physiological and audio data, but no video). Among the 65 participants, 62 opted for option and opted for option B (2 women and 1 man).
Each participant was recorded in a single session, lasting approximately 35 minutes. They were instructed not to smoke, intake caffeine, or exercise 3 hours before the experiment. At the beginning of each session, they were introduced to the purpose and content of the study. The experiments are conducted entirely in English. The experimental protocol was identical for all participants, and the experimenter was always present in the room during the recording.
3.2.2 Dataset Composition
Following data collection, we split each recorded session into individual tasks: one 3 minutes breathing recording (block 1), recordings corresponding the watching of the video clips of respectively 2 and 3 minutes (block 2). separate 1-minute recordings of the interactive tasks (block 3), and 2 minute and 30 seconds long relaxation recording (block 4). As the guided breathing, the video clips and the relaxation parts do not carry meaningful audio, the audio part of the dataset consists of the 7 talking tasks only. During the acquisitions, due camera malfunctions, video and audio recordings were damaged. More information about the available tasks for each modality can be found in Appendix A. After splitring, StressID is composed of 711 distinct annotated recordings of the physiological modalities, 587 annotated videos, and 385 annotated audio recordings. In total, the final task-split dataset amounts to approximately 19 hours of annotated physiological data, 15 hours of annotated video data, and 6 hours of annotated audio data, thus amounting to more than 39 hours of data in total. Each task is identified in the dataset by subjectname. task, where the task names are as described in Section 3.I.I This convention facilitates different types of analyses, whether subject-specific or specifee
3.3 Data Annotation
Each task annotated using the answers to self-assessment questions. The first 2 questions establish the participants' perceived stress and relaxation levels on 0-10 scale. Additionally, they answer the SAM [9] to assess their valence and arousal on a 0-10 scale. Research suggests relaxation and stress conditions can be described in different quadrants of the arousal-valence space. For instance, high arousal and negative valence are characteristics of emotional stress induced by threatening stimuli (15),
while low arousal and positive valence are characteristics of a calm and relaxed state (37].
The distributions of the StressID self-assessments are reported Figure 3 distributions highlights a positive correlation between stress and arousal, as well as relax and valence. This suggests that across subjects and tasks, a high arousal is associated with a higher level of stress, and a positive valence corresponds to a higher level of relaxation. In addition, the marginal distributions of stress and relax ratings (Figure 3) highlight a balance in the perceived stress and relaxation levels of the participants across the whole experiment, suggesting that the experimental protocol of StressID can arouse proportional instances of stress and relaxation. Furthermore, the distribution of arousal is significantly skewed towards a high rating across the dataset, while valence is centered around a neutral value, highlighting the ability of the protocol to create a high involvement in the participants and elicit strong responses. An extended analysis of the self-assessment distributions analyses can be found in Appendix E.
The analysis of the
We propose 2 discrete labels that can be used to train supervised models: a 2-class label and a 3-class
one. The 2-class label is computed using the stress self-assessment of each task by splitting the 0-10
scale at 5. Precisely, tasks with self-assessment of stress below 5 are considered not stressed (0)
while tasks with self-assessmment equal or above are stressed (1). The 3-class label is based on
the results outlined by [15] 37,. which are in line with the observations drawn from Figure 3 It
allows the prediction relaxed vs. neutral vs. stressed. We considered subject to be relaxed (0)
for task where they reported a valence rating above 5, an arousal rating below 5, and perceived relaxation rating above 5. Similarly, we label tasks with arousal levels above 5, valence levels below 5, and perceived stress levels above 5 as stressed (2), and neutral (1) otherwise.

---

count
count
30
35
25
20
25
stress
15
relax
15
10
10
arousal
valence
(a) Sress/Arousal
(b) Relax/Valence
Figure 3: Distribution the self-assessment answers. (left) Joint and marginal distributions of stress and arousal. (right) Joint and marginal distributions of the relax and valence ratings.
4 Baselines
We implement several unimodal and multimodal baselines combining features extracted from video, audio, and physiological inputs. We train the models to perform 2-class classification, i.c. binary discrimination between stressed and not stressed, as well as 3-class classification. In all the ex- periments, generate 10 random splits, using 80% of the tasks for training, and 20% for testing for each split. The results are averaged over the 10 repetitions. To ensure robustness to potential assessed using the weighted F1-score
imbalance resulting from the train-test splits, the results and the balanced accuracy on the test data. The full list of extracted features, additional experiments, model hyperparameters, and training details are reported in Appendix The implementation of all the baselines can be found at https:: h htt//gittaa thub. com/ homn/robustmllm eurecom/ stressID
4.1 Unimodal Baselines
Each unimodal baseline is trained and tested on all available tasks of the corresponding modality, i.c. 715, 587, and 385 tasks respectively for the physiological, video, and audio baseline. In the following. we describe the baselines for each modality. The obtained results are reported in Table 2
Physiological Signals. In line with the literature stress recognition from physiological signals 55
20 21., we propose baseline including pre-proced pre-processing of the signals, feature extraction, and classification. In a first step, the ECG, EDA, and respiration signals are filtered with Butterworth filters to reduce high-frequency noise and baseline wander. Then, 35 ECG features, 23 EDA, and 40 respiration features extracted. These include HRV features in the time domain including the number of R to R intervals (RR) per minute, the standard deviation of all NN intervals (SDNN), the percentage of successive RR intervals that differ by more than 20ms and 50ms (pNN20 and PNN50), or the root mean square of successive RR interval differences (RMSSD), as well as frequency-domain and non-linear HRV measures. We have extracted statistical features of the Skin Conductance Level (SCL) and Skin Conductance Response (SCR) components of the EDA, including the slope and dynamic range the SCL, along with time domain features including the number of SCR peaks per minute, the average amplitude of the peaks, and average duration of SCR responses. In addition, we have extracted Respiration Rate Variability (RRV) features in the time and frequency domains. The resulting handcrafted (HC) features are then classified using classical Machine Learning (ML) algorithms: a Random Forests (RF) classifier, Support Vector Machines (SVM), and a Multi-L.ayer Perceptron (MLP) with hyperparameters chosen by Cross-Vaas Validation (CV).

---

Table 2: Performances of unimodal baselines for the classification of stress. Each baseline is trained and tested on all available tasks of the corresponding modality.
2-class
3-class
Baseline
Fl-score
Accuracy
FI-score
Accuracy
Physio. HC features + RF
0.73 + 0.02
0.72 + 0.03
0.55 + 0.04
0.56 = 0.03
Physio. HC features SVM
0.71 + 0.02
0.71 + 0.02
0.59 + 0.04
0.59 + 0.03
Physio. HC features + MLP
0.70 + 0.03
0.70 = 0.03
0.54 = 0.04
0.53 0.04
AUs + kNN
0.70 + 0.04
0.69 + 0.04
0.54 = 0.05
0.53 + 0.05
AUs SVM
0.69 + 0.04
0.69 + 0.04
0.55 0.05
0.54 0.04
AUs + MLP
0.70 + 0.03
0.70 + 0.03
0.55 + 0.03
0.55 + 0.03
Audio HC features + KNN
0.67 + 0.06
0.60 + 0.05
0.53 + 0.04
0.52 + 0.04
Audio HC features + SVM
0.61 0.06
0.54 + 0.03
0.53 - 0.08
0.48 0.04
W2V 2.0 classifier
0.70 + 0.02
0.66 + 0.03
0.56 + 0.04
0.52 + 0.04
Video Data. We propose a baseline employing Action Units (AU) and eye gaze for the classification
of stress. AUs are commonly used as features stress recognition applications [2] 27 2 They are
fine-grained facial muscle movements 18) each relating to subset of extracted facial landmarks [40] Each AU is described in two ways: presence, the AU is visible the face, and intensity, indicating how intense the AU is on 5-point scale (minimal to maximal). After downsampling the recordings to 5 frames per second, we use the OpenFace library [8] to extract eye gaze and AUs from each video frame. We extract the following AUs: 1, 2, 4, 5, 6. 7, 9, 10, 12. 14, 15, 17, 20, 23, 25, 26, 28, and 45. As eye gaze features, we use two gaze direction vectors computed individually for each eye by detected pupil and eye location. The averages and standard deviations of each AU and eye gaze directions are computed across time frames. The resulting 84-component vector is used as input to several models: k-Nearest Neighbors (kNN) algorithm, an SVM, and MLP with 4 layers of width 256. In line with (27). the number of layers and layer width of the MLP are chosen by CV in (2,3,4) and (64, 128, 2 respectively. We use ReLU activation and the MLP is trained for 100 epochs with cross-entropy loss optimized using Adam 31 with an initial learning rate of 1e-3.
Audio Data. We propose two baselines for speech signals: the first employs HC features and ML algorithms, and the second is built on the Wav2Vec 2.0 (W2V) model [49 6). Both techniques involve downsampling from the original 32 kHz audio to 16 kHz, and the application of amplitude- based voice activity detection (VAD) 32 prior to feature extraction to eliminate non-speech segments.
The first baseline relies on plethora of specific audio features (46 4 widely used in the literature on emotion recognition from speech 35). These include Mel Frequency Cepstral Coefficients (MFCCs) and their first and second derivatives, which characterize the short-term power spectrum and its dynamics. The spectral centroid, bandwidth, contrast, flatness, and roll-off, which together provide rich statistical representation of the spectral shape. Harmonic and percussive components are also extracted, with tonal centroid features being computed for the harmonic component. The zero- crossing rate is a simple measure of the rate of sign changes; the rate of zero-crossings relates directly to the fundamental frequency of the speech signal. Last, we include tempogram ratio features [39] which represent local rhythmic information. We compute the mean and standard deviation over time for all features, thereby resulting in feature vectors for each, which are then concatenated to form comprehensive feature vector of 140 components, and used as input for MI. algorithms.
The second baseline employs a large, pre-trained W2V model. The W2V 2.0 model produces features capturing a wealth of information relevant diverse tasks including emotion recognition [11 50 [4]. Features are extracted every 20 ms and averaged over time to obtain a single 51 513-component embedding per utterance, and are then classified using linear classification layer optimized with Adam, cross- cross-entropy loss, and an initial learning rate of 1e-3, until convergence.
4.2 Multimodal Baselines
Multimodal baselines that combine the features extracted from all sources are evaluated on the tasks that feature all modalities only, i.e. 370 tasks, to avoid learning with severely missing values. This subset of StressID is composed of talking tasks exclusively, i.e. all tasks without the audio modality are excluded. In this setting, the dataset presents strong imbalance in the labels (70%
8

---

Table 3: Performances of multimodal baselines for the classification of stress, compared to unimodal models, All baselines are trained and tested only on tasks featuring all modalities, i.e. 370 tasks.
2-class
3-class
Baseline
Fl-score
Accuracy
Fl-score
Accuracy
Physiological only
0.66 0.05
0.58 + 0.04
0.50 = 0.05
0.48 = 0.06
Video only
0.67 0.03
0.62 + 0.04
0.58 0.05
0.56 = 0.05
Audio only
0.67 + 0.04
0.62
0.56
0.54 = 0.06
Feature fusion SVM
0.64 + 0.09
0.56
0.55
0.51 = 0.05
Feature fusion MLP
0.66 + 0.04
0.01
0.31
0.51 = 0.07
Feature fusion DBN
0.58 0.06
0.00
0.00
0.32 0.04
SVM Sum rule fusion
0.72 0.05
0.04
0.05
0.58 0.07
SVM Product rule fusion
0.71 + 0.05
0.63 0
0.61 0.05
0.56 0.07
SVM Average rule fusion
0.72 0.05
0.65 + 0.05
0.63 - 0.05
0.58 0.07
SVM Maximum rule fusion
0.72 0.05
0.64 + 0.05
0.61 0.06
0.57 = 0.07
stress). We Minority Over-sampling Techniques (SMOTE)
13
balance the training set in each
of the 10 repetitions, and leave the test sets untouched.
We propose fusion models combining all features using the most prominent fusion methods in the literature: fcature-level and decision-leve fusion 38). For feature-level fusion, unimodal HC features are combined into single high-dimensional feature vector, used as input for learning algorithms. Similarly to 22 [2], we evaluate feature-level fusion combined with SVM, MLP classifiers, and Deep Belief Networks (DBN). For decision-level fusion, following [57 45 we train independent SVMs for each modality using the HC features as input, and integrate the results of the individual classifiers at the decision level, i.e. the results combined into single decision using ensemble rules. The results for all multimodal baselines for the 2-class and 3-class classification are reported in Table To ensure fairesss in the comparison, the multimodal baselines are evaluated against best-performing HC and MI.-based unimodal baselines (Section I. trained on the subset of the 370 tasks featuring all modalities. Additional results for all other modality combinations reported in Appendix F.2.2.
Limitations
First, this dataset is recorded in controlled environment specifically designed to elicit responses. Experiments conducted in laboratory settings do not take into consideration the external factors that contribute to the psychological mental state of participants and typically assume stress reaction is an isolated occurrence. In reality, human emotions are complex and are influenced by combinations of factors. In addition, the process of attaching electrodes to the participants may be stressful in itself. Therefore, the signals recorded in this setting are not necessarily representative of real-life situations. In consequence, although models built the StressID dataset can learn to reliably recognize a response to stress induscnn stimuli, the discrimination between positive and negative, or short-term and long-term stress is more sensitive task. Second, relying self-assessed scales for data annotation is a participant-subjectivee process, and can lead to bias in subsequent analyses. another. Nevertheless,
Perception of stress and relaxation can vary lot from one participant analyses described in Section (3.3 highhlght a coherent distribution of the self-reported annotations across participants and the whole experiment. Third, although all participants recruited for the study are proficient in English, the act of speaking English itself can be stress-ing dtress-inducing for non- native speakers. Fourth, the audio component of the dataset suffers from an uneven distribution of labels, as the verbal tasks are associated with higher levels of stress. Fifth, Stress suffers from missing modalities for some participants. Finally, StressID presents a gender imbalance representative of the female/male ratio in STEM studies and workforce [19]. This is a limitation StressID shares with competitor datasets 33
36 27), and a common issue in human data collection, in general [17 41 Additional experiments sensitising users to the effect of gender bias and demonstrating how Stress ID can effectively be used to build equitable applications, are reported in Appendix F.2.3.

---

Ethical Considerations and Dataset Accessibility
The recording and usage of human activity data are associated with ethical considerations. The StressID project is approved by the ethical committee of the Université Cote d' Azur (CER). The experiments have been conducted under agreement n² 2021-033 for data collection, and no 2023-016 for the publication of the dataset. The participants explicitly consent to the recording of their session, the dataset creation, and its release for research purposes following General Data Protection Rules (GDPR). The personal information (sex, age, education), and the acquired physiological and audio signals are pseudonymized, and an alphanumeric code is given for each participant. Video data can not be anonymized and is treated as sensitive data.
Given the identifying nature of the facial videos, the dataset is made accessible through open credentialized access only, for research purposes. Users are required to sign an end-user license agreement to request the data. Once validated a secured set of credentials granted to access to the dataset. The dataset uses proprietary license for research purposes and it is hosted on Inria servers using storage intended for long-term availability. The code uses an open-source license. We are aware that despite all precautions, the dataset can be misused by bad-intentioned users. The data and the code for the baselines are available at htttps://rrojess inria. fr fr/stressid/
Lastly, systems that use the dataset for modeling and understanding the mechanisms of human stress conditions need to be aware of the potential imbalance in representation in the dataset. Participants for the data collection were included in our dataset without restrictions on gender, race, age, or education level instead favoring sample size.
7 Conclusion
We present StressID, a dataset for stress identification featuring three categories of data modalities and three different types of stimuli. The experimental protocol designed to collect the Stress ID dataset is easy to replicate and can be adapted to additional sensors or stressors. The equipment used for the data collection is affordable, and the selected devices guarantee low noise in the recordings.
The multimodal nature of StressID offers large set of possible uses cases and applications. On one hand, diverse modalities carry complementary information that can be jointly exploited: video and audio capture the behavioural component of emotions, the reactions that are visible from outside, while the physiological signals capture valuable internal states not visible on camera such as cardiac activity. or skin sweating. By providing access to multiple synchronized modalities, StressID enables cross-modal analyses that have the potential to improve the understanding of the relationships between video, audio, and physiological responses to stress. On the other hand, the dataset design also offers the possibility to develop models of different natures, by focusing on single modality. Moreover, it allows a wide range of applications, including subject-specific, or task-specific studies.
StressID dataset can contribute to advance research in multiple fields. First, it has the potential to improve the understanding of the sources, demographics, and both physical and physiological mechanisms of stress responses. It is designed for the development of reliable algorithms for stress identification that can improve the quality of life of our society by helping prevent stress-related issues. For instance, early stress recognition can be beneficial for people suffering from neurological or developmental disorders with emotion deregulation, such as autism, for whom the increase of stress can cause disruptive behaviors. Second, Stress ID can help improve affect understanding, as it offers the possibility to analyze and understand the correlation patterns between the distributions of perceived stress and emotion, how these correlations relate to different categories of stimuli, or how they impact subsequent stress and emotion recognition algorithms. Finally, StressID is useful to the machine learning and deep learning communitics as well, as it can be used to further evolve multimodal learning algorithms, to develop strategies for learning with unevenly represented modalities, or to study how to make algorithms learning with human data more reliable.
To foster reproducibility, StressID also offers a set of baseline experiments. Although the proposed models focus on predicting two discrete labels designed to illustrate the predictive potential of our dataset, they represent a good starting point for future work, to which researchers and developers can benchmark their work. this context, a natural extension of this work would be the implementation of web service that tracks and centralizes the performances of models developed using StressID.
10

---

Acknowledgments and Disclosure of Funding
HC. VS, FB and MAZ are supported by the French government, through the 31A Côte d' Azur Investments in the Future project managed by the National Research Agency (ANR) (ANR-19- P3IA-0002). MP, NE and MT are supported by the ANR RESPECT Project (ANR-18-CE92. LMF has been partially supported by the Ville de Nice and the French government, through the UCAJEDI Investments in the Future project managed by the ANR MANR-15-IDEX-. and by PNRR- Investment 1.5 Ecosystems of Innovation, Project Tuscany Health Ecosystem (THE), Spoke 3 "Advanced technologies, methods, materials and heath analyties" CUP: 153C2200078001..
References
[1] Naveed Ahmed, Zaher Al Aghbari, and Shini Girija. A systematic survey on multimodal emotion recognition using learning algorithms. Intelligent Systems with Applications, 17:200171, 2023.
[2] Jonathan Aigrain, Michel Spodenkiewicz, Severine Dubuisson, Marcin Detyniecki, David Cohen, and Mohamed Chetouani. Multimodal stress detection from multiple assessments, IEEE Transactions on Affective Computing. 9()):411-506, 2016.
[3] Andrew P Allen, Paul J Kennedy, Samantha Dockray, John F Cryan, Timothy G Dinan, and Gerard Clarke. The trier social stress test: principles and practice. Neurobiology of stress, 6:113-126, 2017.
[4] Alo Allik, György Fazekas, and Mark B. Sandler. An ontology for audio features. In Interna- tional Society for Music Information Retrieval Conference, 2016.
[5] Aamir Arsalan, Syed Muhammad Anwar, and Muhammad Majid, Mental stress detection using data from wearable and non-wearable sensors: review. arXiv preprint arXiv:2202.0303.. 2022.
[6] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations, Advances in neural information processing systems, 11112244-12464 2020.
[7] Anjana Bali and Amteshwar Singh Jaggi. Clinical experimental stress studies: methods and assessment. Reviews in the Neurosciences, 26(5):555 579, 2015.
[8] Tadas Baltrusaitis, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency. Openface 2.0: Facial behavior analysis toolkit. In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), pages 19.66. IEEE, 2018
[9] Margaret M Bradley and Peter J Lang. Measuring emotion: the self-assessment manikin and the semantic differential. Journal of behavior therapy and experimental psychiatry, 25(1):49-59, 1994.
[10] J Douglas Bremner. Traumatic stress: effects on the brain. Dialogues in clinical neuroscience, 2022.
[11] Fabio Catania. Speech emotion recognition in italian using wav2vec 2.0 and the novel crowd- sourced emotional speech corpus emozionalmente. 2023.
[12] Valentina Chaparro, Alejandro Gomez, Alejandro Salgado, Lucia Quintero, Natalia Lopez, and Luisa F Villa. Emotion recognition from eeg and facial expressions: a multimodal approach. In 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pages 530-533. IEEE, 2018
[13) Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321- 357, 2002.
[14] Li-Wei Chen and Alexander Rudnicky. Exploring wav2vec 2.0 fine tuning for improved speech emotion recognition. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
11

---

[15] Sven-Ake Christianson, Emotional stress and eyewitness memory: critical review. Psycholog- ical bulletin, 112(2):284, 1992.
[16] Joel E Dimsdale. Psychological stress and cardiovascular disease. Journal of the American College of Cardiology, 51(13):1237-1246, 2008.
[17] Anila M D' D'Mello, Isabelle R Frosch, Cindy E Li, Annie L Cardinaux, and John DE Gabrieli. Exclusion of females in autism research: Empirical evidence for a "leaky" recruitment-to- research pipeline. Autism Research, 15(10):1929-194) 2022.
[18] Paul Ekman and Wallace V Friesen. Facial action coding system. Environmental Psychology & Nonverbal Behavior, 1978.
[19] UNESCO Institute for Statistics, Women in science: Fact sheet no. 55. https://uis / unesco /esstee//aafallttlle///////eeo
wwmen in- science 2019-en. .ddf
2019. Accessed: 2023-10-19.
[20] Shruti Gedam and Sanchita Paul. review mental stress detection using wearable sensors and machine learning techniques. IEEE Access, 9:84045-84066, 2021.
[21] Giorgos Giannakakis, Dimitris Grigoriadis, Katerina Giannakaki, Olympia Simantiraki, Alexan- dros Roniotis, and Manolis Tsiknakis. Review psychological stress detection using biosignals. IEEE Transactions on Affective Computing, 13(1):440 4600 2019.
[22] Giorgos Giannakakis, Mohammad Rami Koujan, Anastasios Roussos, and Kostas Marias. Auto- matic stress detection evaluating models of facial action units. 2020 15th IEEE international conference automatic face and gesture recognition (FG 2020), pages 728-733. IEEE, 2020.
[23] Auriane Gros, Emmanuelle Chapoulie, Remy Ramadour, Vincent Robert, Julie de Stoutz, Stephane Guetin, Damoun Chaïma, Emeline Wyckaert, Valeria Manera, Philippe Robert, et al. @el@ Sensory and virtual immersion to reduce the anxiety of patients consulting for the first time in nice memory center. Alzheimer's 's and Dementia, 13(7):P609-PP6 2017.
[24] Sandra G Hart. Nasa task load index (Ilx). 1986.
[25]
Jennifer Anne Healey. Wearable and automotive systems for affect recognition from physiology. PhD thesis, Massachusetts Institute of Technology, 2000.
[26] Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes multi-modal learning better than single (provably). Advances in Neural Information Processing Systems, 34:10944-10956, 2021.
[27] Mimansa Jaiswal, Cristian-Paul Bara, Yuanhang Luo, Mihai Burzo, Rada Mihalcea, and Emily Mower Provost. Muse: multimodal dataset of stressed emotion. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1499-1510, 2020.
[28] Apichart Jaratrotkamjorn and Anant Choksuriwong. Bimodal emotion recognition using deep belief network. In 2019 23rd International Computer Science and Engineering Conference (ICSEC), pages 103-109. IEEE, 2019.
[29] Tzyy-Ping Jung, Terrence J Sejnowski, et al. Utilizing deep learning towards multi-modal bio-sensing and vision-based affective computing. IEEE Transactions on Affective Computing, 13(1):96-177, 2019.
[30] Hye-Geum Kim, Eun-Jin Cheon, Dai-Seg Bai, Young Hwan Lee, and Bon-Hoon Koo. Stress and heart rate variability: a meta-analysis and review of the literature. Psychiatry investigation, 15(3):235, 2018
[31] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015.
[32] Tomi Kinnunen and Haizhou Li. An overview of text-independent speaker recognition: From features to supervectors. Speech communication. 52(1):12-40, 2010.
12

---

[33] Saskia Koldijk, Maya Sappelli, Suzan Verberne. Mark A Neerincx, and Wessel Kraaij The swell knowledge work dataset for stress and user modeling research. In Proceedings of the 16th international conference multimodal interaction, pages 291-298, 2014.
[34] Eun-Hyun Lee. Review of the psychometric evidence of the perceived stress scale. Asian nursing research, 6(4):121-127, 2012.
[35] Surendra Malla, Abeer Alsadoon, and Simi Kamini Bajaj. A dfc taxonomy of speech emotion recognition based on convolutional neural network from speech signal. In 2020 Sth International Conference Innovative Technologies in Intelligent Systems and Industrial Applications (CITISIA), pages 1-10, 2020.
[36] Valentina Markova, Todor Ganchev, and Kalin Kalinkov. Clas: A database for cognitive load, affect and stress recognition. 2019 International Conference on Biomedical Innovations and Applications (BIA), pages 1-4. IEEE, 2019.
[37] Maria D McManus, Jason T Siegel, and Jeanne Nakamura. The predictive power of low-arousal positive affect. Motivation and Emotion, 43:130- 1440, 2019.
[38] Asif Iqbal Middya, Baibhav Nag, and Sarbani Roy. Deep learning based multimodal emotion recognition using model-level fusion of audio-visual modalities. Knowledge-Bern Systems, 244: 188580, 2022.
[39] Geoffroy Peeters. Rhythm Classification using spectral rhythm patterns. In ISMIR, pages London, United Kingdom, September 2005. cote interne IRCAM: Peeters05b.
[40] Nazil Perveen and Chalavadi Krishna Mohan. Configural representation of facial action units for spontaneous facial expression recognition in the wild. In VISIGRAPP (4: VISAPP), pages 93-102, 2020.
[41] Ana-Catarina Pinho-Gomes, Jessica Gong, Katie Harris, Mark Woodward, and Cheryl Carcel. Dementia clinical trials over the past decade: are women fairly represented? BMJ Neurology Open, 4(2), 2022.
[42] PLUX wireless biosignals S.A. Electrocandiography (ECG) Sensor User Manual, 2020.
[43] PLUX wireless biosignals S.A. Electrodermal Activity (EDA) Sensor Datasheet, 2021.
[44] PLUX wireless biosignals S.A. Respiration (PZT) Sensor User Manual, 2021.
[45] K Prasada Rao, MVP Chandra Sekhara Rao, and N Hemanth Chowdary. An integrated approach to emotion recognition and gender classification. Journal of Visual Communication and Image Representation, 60:339-345, 2019.
[46] Md. Sahidullah, Tomi Kinnunen, and Cemal Hanilçi. A comparison of features for synthetic speech detection. In INTERSPEECH 2015, 16th Annual Conference of the International Speech Communication Association, Dresden, Germany, September 6-10, 2015, pages 2087-2091. ISCA, 2015.
[47] Alexandre Schaefer, Frédéric Nils, Xavier Sanchez, and Pierre Philippot. Assessing the effec- tiveness of large database of emotion-eliciting films: A new tool for emotion researchers. Cognition and emotion, (177):1153-1177 2010.
[48] Philip Schmidt, Attila Reiss, Robert Duerichen, Claus Marberger, and Kristof Van Laerhoven. Introducing wesad, a multimodal dataset for wearable stress and affect detection. In Proceedings of the 20th ACM international conference on multimodal interaction, pages 400-408, 2018.
[49] Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vee: Unsupervised pre-training for speech recognition, 2019.
[50] Mayank Sharma. Multi-lingual multi-task speech emotion recognition using wav2vec 2.0. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6907-6911. 1. IEEE, 2022.
13

---

[51] Charles D Spielberger. Fernando Gonzalez- Senzalle-RRegosa, Angel MartinezUUrutia, Luiz FS Nata- licio, and Diana S Natalicio. The state-trait anxiety inventory. Revista Interamericana de Psicologia//nteramrrian journal of psychology, 5(3 & 4), 1971.
[52] Herman JM Steeneken and John HL Hansen. Speech under stress conditions: overview of the effect on speech production and on system performance. In 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No. 99CH36258), volume 4, pages 2079-2082. IEEE, 1999.
[53] J Ridley Stroop. Studies of interference in serial verbal reactions. Journal of experimental psychology, 18(6):643, 1935.
[54] Salah Taamneh, Panagiotis Tsiamyrtzis, Malcolm Deosta, Pradeep Buddharaju, Ashik Khatri, Michael Manser, Thomas Ferris, Robert Wunderlich, and Ioannis Pavlidis. A multimodal dataset for various forms of distracted driving. Scientific data, 4(1):1-21, 2017
[55] Thi-Dung Tran, Junghee Kim, Ngoc-Huynh Ho, Hyung-Jeong Yang, Sudarshan Pant, Soo- Hyung Kim, and Guee-Sang Lee. Stress analysis with dimensions of valence and arousal in the wild. Applied Sciences, 11 ())5194, 2021.
[56] David Watson, Lee Anna Clark, and Auke Tellegen. Development and validation of brief measures of positive and negative affect: the panas scales. Journal of personality and social psychology, 54(6):1063, 1988.
[57] Fen Xu and Zhe Wang. Emotion recognition research based on integration of facial expression and voice. In 2018 11th International Congress Image and Signal Processing, BioMedical Engineering and Informatics ( (CISP-BMEI) pages 1-6. IEEE, 2018.
14


Reviews:
[{"review_id": "P6b2EnoQva", "reviewer": "Reviewer_9RXE", "summary": "", "questions": "", "limitations": "The limitations are well described with the exception of a few noted problems, which are explained above in the opportunities for improvement.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The contribution of this work will help in the building of robust and reliable, frameworks for stress recognition. The multimodal nature of the data is a strength. Further, the authors include an easy to reproduce experimental protocol such that more data could be collected in the future to augment this data set. The authors have also thought through how the data can be used and demonstrate some examples of stress detection models using this data in practice.\n\nOne major strength is the combination of the two types of labels: the stimulus label, as well as the feelings self-reports. The seven interactive stressors seem to be well validated methods that are based in the literature, which is a strength.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "TjDj5ihkaP", "reviewer": "Reviewer_wbdk", "summary": "", "questions": "", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "(1) The data collection focused on socially significant stress-inducing tasks, gathering diverse modalities of data along with corresponding self-assessment labels.\n\n(2) Detailed explanations were provided regarding the data construction, instruction, and maintenance methods.\n\n(3) Limitations and ethical considerations were well addressed in the study.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "eTTyrAg889", "reviewer": "Reviewer_QzsT", "summary": "", "questions": "", "limitations": "The authors openly discussed the limitation of their work.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The data collection process is well-designed in general. I appreciate the authors' effort in collecting such a dataset.\n- The paper has a very clear structure. The writing has high quality and is very easy to follow.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "nTHjUw5xRQ", "reviewer": "Reviewer_jqtB", "summary": "", "questions": "", "limitations": "1)The use of a multimodal approach to stress detection, which involves attaching a number of electrodes to the subject, could potentially cause stress in itself. This is because the sensors may be uncomfortable or restrictive, and the subject may feel self-conscious or anxious about being monitored. Additionally, the act of attaching the sensors may itself be stressful, as it may remind the subject of a medical procedure or other stressful event. This dataset cannot be related to real world scenarios.\n\n2)When I tried to download the dataset...I am being asked lot of questions...Is the dataset available for download\n\n3)Why the need to record so many signals..seems like overkill\n\n4)Table 2 is not clear…why are tasks number varying physiolo(715), video(605) and so on\n\n5)Why is multimodal classification low..Multimodal classification means all recording  signals are used for classification right?\n\n\n", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The given dataset captures multimodal data related to stress, including audio, video, and physiological signals. These different types of data provide complementary information that can be used to build a more complete understanding of stress.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "FWAY7yIFSH", "reviewer": "Reviewer_qPpi", "summary": "", "questions": "", "limitations": "This article describes the Limitations, which is good. \n\nThe dataset was collected in a carefully arranged laboratory environment, with the stress levels of the subjects controlled by induction materials provided by the experimental protocol, and data annotation determined by subject self-reporting on scales. Therefore, this dataset mainly annotates short-term stress, while stress levels may also be influenced by both short-term and long-term stress; this dataset cannot assess long-term stress levels. The induction materials may have different effects on each individual, and the subjects' reports of their stress levels are subjective; this subjectivity may affect annotation accuracy.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The dataset is not specifically designed for a particular scenario, and it has strong versatility. \n\n2. The dataset includes video and audio data, synchronously collected physiological signals such as ECG, EDA, RR. In addition, derived indicators based on physiological signals were calculated, such as HRV. These indicators play an important role in distinguishing stress levels; therefore the modalities of the dataset are comprehensive. \n\n3. The annotation of the dataset uses the SAM method with participants self-reporting by answering four questions. The authors then provide binary and ternary labels based on participants' responses which expand the application scenarios of this dataset. \n\n4. The paper presents some baseline models for multimodal feature fusion that serve as good examples for algorithm development.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper provides a new multimodal dataset for the important and challenging area of stress identification. The reviews were mixed in terms of whether the paper was of sufficient quality. The set of tasks used for providing stress are novel and interesting, especially given the context of the population. In light of the overall comments and understanding the important context in which the dataset was collected, I have decided to recommend acceptance.

The reviewers expressed doubt about the novelty of the dataset given the evidence from Table 2 of the paper. 
The rewrite of the "comparison of the state of the art" section is still unfortunately not clear e.g. "Indeed, StressID employs diverse and carefully selected tasks, and rather than relying on task-based ground truths,...". However, given the confidence of other authors and also the discussion I make below, I think the work should be accepted providing the changes below are made to the presentation of the work.

Regarding the gender imbalance as highlighted by reviewer 9RXE. If the aim of the dataset is to train models that can learn stress, anything learned from this dataset could be extremely biased towards males. There is some arguments given by the authors related to the origins of the project funding being about students and workers in STEM environments. This is fine. However, it is not reflected obviously back in the paper presentation at the moment and I think this is problematic for the spirit of the datasets and benchmarks track. It is important to highlight novel application specific demands that inspire the collection of new datasets. 

I think that designing the dataset and presenting the dataset to focus on the goals of the project more directly would make the paper stronger. As it stands now the motivation of the design of the stimuli and how it differs from related datasets is rather confusing. If there are concerns about the dataset being too specific, there are always opportunities to specify in the writing of the paper how this dataset might still be useful in a more generalised context.

Finally, please address in presentation of the paper either in the main text or supplementary: e.g. what new research questions are enabled by the data wrt the ML community, application domain, as well as for affect understanding?
Decision: Accept (Poster)