社団法人 電子情報通信学会
THE INSTITUTE OF ELECTRONICS,
INFORMATION AND COMMUNICATION ENGINEERS

信学技報
TECHNICAL REPORT OF IEICE.

Detecting Distress Variations Using Multimodal Data Obtained
through Interaction with A Smart Speaker
Chingyuan LIN† , Yuki MATSUDA†,†† , Hirohiko SUWA†,†† , and Keiichi YASUMOTO†,††† Nara Institute of Science and Technology 8916-5 Takayama-cho, Ikoma, 630–0101 Japan†† RIKEN Center for Advanced Intelligence Project 1-4-1, Nihonbashi, Chuo-ku, Tokyo, 103–0027 Japan
Abstract Mental health signiﬁcantly aﬀects people, with excessive stress potentially causing depression, low productivity,and suicidal thoughts. It can also harm physical health, impacting appetite and sleep, and may lead to other diseases. In mostcases, individuals do not notice stress buildup until their health severely deteriorates. Thus, daily monitoring of stress levels isessential. In this study, we aim to realize a method to estimate people’s distress levels in everyday life through conversation witha smart speaker. We set up a smart speaker in the bedrooms of participants to simulate a home environment and recorded theirinteractions with it using a webcam. These recordings allowed us to analyze facial expressions, voice, and heart rate data. We
processed these features and predicted levels of Happiness, Depression, and Anxiety. Participants completed questionnaires
using the Depression and Anxiety Mood Scale (DAMS) after each session, providing emotion labels with scores from 0 to 18.
In a 14-day experiment involving seven participants aged 22-24, the MAE for Happiness, Depression, and Anxiety levels were
2.04, 2.59, and 2.31, respectively, while the RMSE for these distress levels were 2.63, 3.20, and 2.91.
Key words distress, happiness, anxiety, depression, stress, audio-visual, heart rate, multimodality, smart speaker, DAMS.

1. Introduction

In our study, we gathered 40-second video recordings of 7 individuals interacting with a smart speaker for 14 days. We extracted

In recent years, the shift from oﬃce-based to remote work has

facial expressions, voice, and heart rate features from these videos.

become more prevalent. By 2023, 12.7% of full-time employees

Using Random Forest and LightGBM Regression Models, we pre-work from home, with 28.2% in a hybrid model. Tokyo has seen

dicted changes in Happiness, Depression, and Anxiety levels. The

over half of its businesses adopt remote work, a signiﬁcant increaselabels, ranging from 0 to 18, were based on the Depression and Anx-

from less than 30% pre-COVID-19, maintaining a rate above 50%

iety Mood Scale (DAMS) questionnaires completed post-recording.

by the end of 2022. While remote work reduces physical healthWe evaluated our models using 10-Fold Cross Validation, Leave-risks associated with outdoor jobs, concerns about the psychologi-

Person-Day-Out Cross Validation and Leave-One-Day-Out Cross

cal well-being of workers have escalated. Issues like psychological

Validation, assessing accuracy with Mean Absolute Error (MAE)

stress and depression are increasingly prominent due to the indoor

and Root Mean Squared Error (RMSE). The lowest MAE scoresoﬃce environment. Psychological conditions, unlike physical ill-for Happiness, Depression, and Anxiety were 2.04, 2.59, and 2.31,

nesses, develop gradually and are harder to detect early on. Despite

respectively, while the lowest RMSEs were 2.63, 3.20, and 2.91.

research on various treatments [1], the subtle nature of mental healthconditions makes early detection challenging, highlighting the needfor early prevention and intervention.

2. Related WorkTo predict the categories of emotions such as stress, anxiety, andThe study focuses on monitoring daily emotional distress indica-happiness, many studies have used diﬀerent methods to improve ac-

tors in users to help them understand their mental state. It utilizes

curacy. Considering the following experiment procedure and model

facial expressions, acoustic information, and heart rate to measuretraining, we categorize these diverse studies based on three distinct

sadness, depression, and anxiety levels in participants. Data is

focal points for discussion. In Section 2.1, we list some of the studies

collected in the participants’ homes to ensure comfort and gather

focusing on the utilization of audio-visual data to predict emotions.

authentic data. Smart speakers, common in households for provid-

Moving on to Section 2.2, we explore how previous studies obtained

ing information and aiding in health and education, will be a central

physiological signal information and how to utilize its representa-

tool in this experiment. These devices, used brieﬂy and frequently

tions. In Section 2.3, we summarize the problems in these related

by various age groups, will be employed to record and analyze com-works and explore suitable research approaches.

munication patterns with participants.
—1—

2. 1 Emotion Recognition Using Visual Data

The detection of emotions in their study is achieved by correlating

Facial expression is one of the features commonly used to ana-

these physiological signals with the arousal and valence data fromlyze emotion or stress levels due to their ability to convey nuanced

this dataset, aiming to classify the aﬀective state of a person accu-

non-verbal cues. Analyzing visual content enhances the depth andrately. Their model demonstrated an impressive accuracy of 0.76 foraccuracy of emotional assessment, making it a valuable resource

Arousal, outperforming other models such as MESAE, traditionalin psychological research. Mohammad Soleymani et al. utilized

DNN, and CNN. Sriramprakash et al. focused on utilizing heart rate

the MAHNOB-HCI database, employed face tracker technology to

and Galvanic Skin Response (GSR) information to extract featuresdetect landmarks from features, and achieved the highest accuracy

indicative of stress levels in working individuals [8]. They appliedon the LSTM model [2]. Dagar et al. introduces needs and appli-

K-Nearest Neighbour (KNN) and Support Vector Machine (SVM)

cations of facial expression recognition, gives a brief introduction

algorithms to classify the extensive open dataset SWELL-KW. Their

towards techniques, applications, and challenges of automatic emo-

results showed accuracies of 66.52% and 72.82%, respectively.

tion recognition system [3]. In their framework, they have adeptly

The studies mentioned above used public datasets with diﬀerent

utilized frames extracted from live streaming, applying an advanced

kinds of sensors to track things like the participants’ heart signals

Grabor feature extraction technique. This is further coupled with aor heartbeat counts. Some of these studies used sensors that peo-sophisticated Multi-Layer Perceptron (MLP) neural network. And

ple had to wear close to their bodies for a long time. It made theMehmet Akif Ozdemir et al. propose a low-cost and functionality

participants uncomfortable and less interested in being part of themethod for real-time classiﬁcation of seven diﬀerent emotions by fa-

experiment, and also complicated research requiring frequent daily

cial expression based on LeNet CNN architecture [4]. They merged

data collection.

3 datasets (JAFFE, KDEF, and their custom dataset) and achieved

2. 4 Research Directions

an accuracy of 96.43% and validation accuracy of 91.81% for the

The issues presented in the above-mentioned related work mainlyclassiﬁcation of 7 diﬀerent emotions through facial expressions byfocus on the experiments and their data collection or environment.

their CNN model-based LeNet architecture.

The following items are the key points that we would like to empha-

2. 2 Emotion Recognition Using Audio Data

size in the experiment of this study:

Audio, much like facial expressions, plays a crucial role in emo-•

Simple experimental conditions

tion and stress level analysis due to its capacity to convey subtle•

A familiar activity space for participants•

Short-time data collection•

Aligning with participants’ daily activities

verbal and non-verbal nuances. For example, Spectrogram [5] visually represent the timbre, pitch, and rhythm of a voice, transform
audio signals into visual data, revealing patterns and intricacies inspoken language that are key to identifying emotional states. This

Regarding the solution of the items shown above, we describe it inmakes it an important tool for the analysis of emotional states andmore detail in the proposed method.

levels. Popova et al. consider and verify a straightforward approachin which the classiﬁcation of a sound fragment is reduced to the

3. Proposed Method

problem of image recognition, and the waveform and spectrogram

The purpose of this study is to enable individuals to monitor

are used as a visual representation of the image [6]. The experimentchanges in their distress levels. To achieve this, we intend to cre-

was done based on the Radvess open dataset, including 8 diﬀerent

ate a system capable of collecting audio-visual data from interac-emotions, and half of which are negative emotions, then used thetions between individuals and smart speakers in their daily lives.

VGG-11 convolutional neural network as the image classiﬁer. The

The ﬂowchart in Fig. 1 illustrates the process, with the environment

result of this experiment was 71% instead of 12.5% accuracy for a

setup as well as data collection at the top, data processing and feature

random choice.

extraction in the middle, and computation of results using various

2. 3 Emotion Recognition Using Physiological Signal

machine learning models at the bottom. We will explain each step

In addition to facial expressions and audio data, physiological sig-in the following subsections.

nals like heart rate variability (HRV) and electrocardiogram (ECG)

3. 1 Assumed Environment and Data Collection

patterns are crucial in predicting the categories of emotions or emo-

The study was conducted in participants’ private rooms, chosen fortional indices. HRV and ECG waveform can detect subtle emotional

their ability to elicit more authentic emotional expressions at home.

changes, often imperceptible to the naked eye. This objective data

This also simpliﬁed the experimental setup by avoiding the need foroﬀers a reliable basis for emotional analysis. To analyze physiolog-

a ﬁxed external location. Participants were given the freedom toical signals obtained from various devices or sensors, Santamaria-

choose the content and timing of their conversations, unlike con-

Granados et al. employed a deep learning approach using a Deep

trolled experiments with designated videos or structured dialogues.

Convolutional Neural Network (DCNN) [7]. Their study focused on

This approach enhanced comfort and authenticity in emotional ex-a dataset of physiological signals, speciﬁcally the AMIGOS dataset.

pression. Smart speakers, useful in health and education [9]〜[11],—2—Fig. 1: Process of Data Analysis
Fig. 2: Vocal Separation

were central to the study, providing daily information and being userfriendly across age groups [12], [13]. A web camera was installednear the smart speaker to capture the participants’ interactions, witheach session lasting 40 seconds to cover around two question-answer
exchanges. This brief recording duration was chosen to maintainparticipant comfort and encourage their willingness to participate.
3. 2 Feature Extraction
In this subsection, we discuss the extraction of features for diﬀerent modal types (face expression, audio, heart rate), respectively.
3. 2. 1 Face Expression Feature
For the face expression, we utilized the OpenFace toolkit, an

Fig. 3: Heart Rate Segmentation

open-source tool for facial behavior analysis presented by Tadas
Baltrusaitis et al., useful for computer vision, machine learning,ing from the smart speaker, retaining only the human vocal data forand aﬀective computing. OpenFace specializes in facial landmark

training OpenSMILE.

detection, head pose estimation, facial action unit recognition, and3. 2. 3 Heart Rate

eye-gaze estimation. For our study, we focused on extracting 6 eye-

Unlike common heartbeat extraction methods using skin-close

gaze and 23 facial action unit features, like Inner Brow Raiser andsensors or smartwatches, our study employs non-contact heart

Blink, from 40-second video clips. Additionally, we calculated therate measurement using remote photoplethysmography (rPPG) [16],mean and standard deviation of these 29 features.

which is more convenient for participants than wearing sensors.

3. 2. 2 Acoustic Feature

We use the Eulerian Video Magniﬁcation (EVM) technique [17] for

For the acoustic features in the video clips, we use the OpenS-

heart rate estimation [18]. EVM, applied to RGB video, ampliﬁes

MILE open-source tool for the extraction presented by Florian Ey-

subtle skin color changes due to blood ﬂow variations, enabling non-ben et al.. It unites feature extraction paradigms from speech, music,

contact, unobtrusive heart rate monitoring. To analyze the heart rate

and general sound events with basic video features for multi-modal

dynamics in a 40-second video using rPPG, we employed a slid-

processing [14]. We choose extended Geneva Minimalistic Acous-ing window method, breaking the video into overlapping 8-second

tic Parameter Set (eGeMAPS) [15] as the feature set, which contains

segments, shown in Fig. 3. This allowed us to observe changes ina total of 88 parameters. Because the webcam captures both thethe heart rate signal over time. We then extracted various statisticalparticipant’s voice and that of the smart speaker during recordings.

features, like standard deviation and root mean square, from theseWe employed an open technique by using Librosa to separate humansegments. A comprehensive list of the 49 features is in Table 1.

voice and smart speaker’s voice. As shown in the spectrogram in

Combining the 58 features from face expression and the 88 features

Fig. 2, there are noticeable distinctions between the waveforms offrom voice, we have a total of 195 feature dimensions.

human voices and the audio from the smart speaker. The latter ex-

3. 3 Machine Learning Model

hibits a more regular waveform pattern, with a frequency that does

This paper discusses using two regression models, including Ran-not dip below 100 Hz. We exclude the segments of sound originat-

dom Forest Regression (RFR) and Light Gradient Boosting Machine—3—

Table 1: Heart Rate Features
Feature (dim) Algorithm or Explanation
Heart rate sequence (9) heart rate sequence
Max & min (2) max,min(Heart rate sequence)
Heart rate range (1) max - min∑𝑛Mean (1)𝑖=1 ℎ𝑟𝑖 /𝑛∑𝑛Mean absolute deviation (1)|ℎ𝑟𝑖 − 𝑀𝑒𝑎𝑛| /𝑛√𝑖=1∑𝑛Root mean square (1)ℎ𝑟 2 /𝑛√∑𝑖=1 𝑖𝑛2Standard deviation (1)𝑖=1 (ℎ𝑟𝑖 − 𝑀𝑒𝑎𝑛) /𝑛
Coeﬃcient of variation (1)𝑆𝑡 𝑎𝑛𝑑𝑎𝑟 𝑑 𝑑𝑒𝑣𝑖𝑎𝑡𝑖𝑜𝑛 / 𝑀𝑒𝑎𝑛

Relative increase𝑖 (8)ℎ𝑟𝑖+1 − ℎ𝑟𝑖

Relative change𝑖 (8)ℎ𝑟𝑖+1 / ℎ𝑟𝑖

Relative increase rate𝑖 (8)(ℎ𝑟𝑖+1 − ℎ𝑟𝑖 ) / ℎ𝑟𝑖
Fig. 4: Experimental Devices

Directional change index𝑖 (8) 1 𝑖 𝑓 ℎ𝑟𝑖+1 > ℎ𝑟𝑖 𝑒𝑙𝑠𝑒 0* 𝑛 means the length of the heart rate sequence.
* ℎ𝑟𝑖 means heart rate in each window.

Besides, we asked participants to wear a Fitbit AltaHR device torecord physiological signals, the purpose of which was to verify(LightGBM), for training and comparison in a context involving mul-

the accuracy of our rPPG model in calculating heart rates after thetimodal features and high dimensionality. RFR is highlighted for its

experiment.

robustness to high-dimensional data, resistance to overﬁtting, ability

4. 3 Scheduleto handle a large number of features, often showing good general-

Participants are actively engaged in conversations with the smart

ization without much data preprocessing. LightGBM is noted forspeaker, ensuring a minimum of two interactions each day. These in-its eﬀectiveness in high-dimensional, nonlinear problems, being ateractions occur both in the morning, shortly after waking up, and in

gradient-boosting tree method that’s also not sensitive to outliers.

the evening, just before bedtime. The communication typically last

4. Experiment

40 seconds, encompassing two rounds of question-and-answer interactions. There are no constraints imposed on the content or topics ofThere were a total of seven Japanese participants in the experi-these interactions, make participants feel closer to their daily lives.

ment. The experiments were conducted in the participants’ respec-

The process spans a duration of two weeks, and at the conclusion oftive dormitories or homes. To ensure that the brightness of the video

each conversation, participants are kindly requested to complete thewould not be so dark that facial expression and heart rate features

Depression and Anxiety Mood Scale (DAMS) questionnaire.

could not be detected, we requested the participants to maintain atleast a certain level of illumination in the room during recording.
4. 1 Device

5. Result
In this section, we begin by comparing the accuracy of our re-

We used a Raspberry Pi 4 microcomputer to install the system, andmote photoplethysmography (rPPG) model against the true values

within it, and employed FFmpeg to record audio-video ﬁle. FFm-obtained from heart rate data recorded by the Fitbit watches worn by

peg is suitable for various audio and video processing tasks, ranging

the participants. Then delve into the comparison of the Random For-

from decoding and encoding to streaming. After each recording,

est Regression model and the LightGBM Regression model in pre-we utilize the PyDrive library to automatically upload ﬁles to thedicting happiness, depression, and anxiety levels. Model evaluation

cloud server. And a Long-Term Evolution (LTE) Dongle device ismethods encompass 10-Fold Cross Validation, Leave-One-Person-for providing internet connectivity for ﬁle uploads. In terms of other

Out Cross Validation (LOPO) and Leave-One-Day-Out Cross Vali-

devices, we chose the Amazon Echo Dot smart speaker, C920n PRO

dation (LODO) for each participant.

HD webcam which supports 1080p high deﬁnition, and installed a

5. 1 Accuracy of Heart Rate

button as the start button for recording. All devices as shown in
Fig.4.

In our method, we applied Eulerian Video Magniﬁcation (EVM) to
perform remote photoplethysmography (rPPG) for estimating heart

4. 2 Questionnaire

rates from video data. We compared our results with the heart rate

We chose Depression and Anxiety Mood Scale (DAMS) [19] asdata from Fitbit AltaHR, which records heart rates every 15 seconds.

the questionnaire in the experiment, which is speciﬁcally designed

Using the Fitbit API, we synchronized the video recording times withto assess the levels of happiness, depression, and anxiety distress.

the Fitbit data to obtain and average the heart rate values. Our heart

It is suitable for daily multiple entries and uses simple adjectives

rate predictions were made using a sliding window technique onfor descriptions, there are three categories of emotional words, eachthe video to segment and average values from diﬀerent parts. We

with three items, and each option ranges from level 0 to level 6.
—4—

Table 2: Accuracy within diﬀerent tolerance ranges
Tolerance Range Accuracy of Heart Rate
<542.79%< 10

67.29%< 15

86.19%

Table 3: MAE / RMSE of Random Forest model
Happiness

Depression

Anxiety

10-Fold CV 2.30 / 2.83 2.85 / 3.40 2.60 / 3.15
LOPO CV2.34 / 2.87 2.71 / 3.36 2.98 / 3.67

LODO CV

2.10 / 2.69 2.59 / 3.20 2.43 / 2.97

Fig. 5: Happiness Level Variations for Subejct 2 in LODO CV

Table 4: MAE / RMSE of LightGBM model
Happiness

Depression

Anxiety

10-Fold CV 2.26 / 2.84 2.97 / 3.55 2.70 / 3.34
LOPO CV

2.61 / 3.24 2.85 / 3.50 2.94 / 3.61

LODO CV

2.04 / 2.63 2.64 / 3.28 2.31 / 2.91

ﬁnally got the results with an MAE of 7.77 and an RMSE of 9.73.
Additionally, we also considered tolerance ranges and presented thecorresponding accuracy for diﬀerent tolerance ranges in Table 2.
5. 2 Prediction Results for Three Distress Levels
In this subsection, we analyze the predicted results for three dis-Fig. 6: Depression Level Variations for Subejct 1 in LODO CV

tress levels using the Random Forest Regression model and theLightGBM Regression model. We also present the results frommultiple subjects in the training set, complicating the identiﬁcation

three diﬀerent cross-validation approaches.

of unique scoring criteria for each individual. Leave-one-day-out

Regarding the prediction results for Happiness, Depression andcross-validation, trained on a single subject’s data, typically yields

Anxiety, Table 3 and Table 4 display the (MAE / RMSE) values

higher accuracy. From Fig. 5 and Fig. 6 provided above, we can ﬁndfrom 10 loops of 10-Fold Cross Validation, Leave-One-Person-Outthat if viewed from a temporal perspective, they can still capture the

Cross Validation and Leave-One-Day-Out Cross Validation for bothoverall trends and ﬂuctuations in distress levels.

the Random Forest Regression model and the LightGBM Regression

6. 2 Comparison of Regression Models
In leave-one-day-out cross-validation, LightGBM Regression

model, respectively.

Model exhibited a higher proportion of lower MAE and RMSE com-

6. Discussion

pared to the Random Forest Regression Model, there are two reasons

Based on Table 3 and Table 4, we can infer that the accuracy of dis-for this result. The ﬁrst one is the lower risk of overﬁtting, compare

tress levels is ranked from highest to lowest as follows: Happiness,to Random Forest, a key advantage of LightGBM is its lower risk ofAnxiety, and Depression. Additionally, for each emotion level, theoverﬁtting, especially in datasets with many features. LightGBM’s

lowest MAE and RMSE are consistently observed during leave-one-leaf-wise tree growth strategy allows it to build deeper and moreday-out cross-validation. And LightGBM has better performancecomplex trees eﬃciently, making it well-suited for high-dimensional

than Random Forest for both Happiness and Anxiety. However, ac-data. Unlike Random Forest, which grows trees level-wise, Light-

curacy still needs to be enhanced. Here we ﬁrst explore why LODO

GBM can capture intricate patterns without signiﬁcantly increasing

CV outperforms the other cross-validation methods and compare

the risk of overﬁtting. The second one is automatic feature selection,

the performance of the two regression models. Subsequently, we

in high-dimensional data, LightGBM eﬃciently identiﬁes and pri-discuss the factors contributing to the overall model accuracy.

oritizes the most inﬂuential features, enhancing model accuracy in

6. 1 Subjective Diﬀerences in Participants’ Perceptions

complex datasets, whereas Random Forest, while eﬀective in many

The DAMS questionnaire employs simple adjective-based ques-

scenarios, often requires post-hoc analysis to determine feature im-tions, leading to greater variations in individual responses. Fig. 5
and Fig. 6 depict distress level variations in the LightGBM model’s

portance.
6. 3 The Accuracy in Heart Rate Prediction

Cross-

According to Table 2, the accuracy of heart rate prediction using

validation methods such as 10-fold and leave-one-person-out involve

Eulerian Video Magniﬁcation (EVM) was not good, with an MAE

leave-one-day-out cross-validation for some participants.

—5—

of 7.77. After reviewing the video data of all participants, we identiﬁed three reasons. The ﬁrst one is the lighting conditions, someparticipants’ rooms tend to have signiﬁcant sunlight exposure in the[6]morning, despite the curtains being drawn. This results in inconsistent light intensities during morning and evening video recordings,
which could potentially aﬀect the accuracy of heart rate detection.

[7]

Another situation is some participants tend to have backlighting during recording, resulting in insuﬃcient lighting on one side of theirfaces. The second one is the distance between the webcam and[8]participants, during the recording process, if participants move backand forth, it will lead to signiﬁcant ﬂuctuations in the predicted heart
rate values. The last one is the limitations of 2D CNN, EVM is common 2D CNN tool for implementing rPPG. However, traditional 2D[9]

CNN approaches lack the capacity to grasp the temporal contextual
aspects of facial sequences. On the contrary, 3D CNN [20] has the[10]capability to simultaneously analyze both the spatial and temporal
attributes of videos, aligning well with the characteristics of rPPG
signals. This is advantageous for remote heart rate measurement.

[11]

7. Conclusion
This study proposed a multimodal approach using imagery, audio, and heart rate data from smart speaker interactions to monitor[12]

users’ Happiness, Depression, and Anxiety levels. Diﬀerent machinelearning regression models were evaluated, with Leave-One-Day-[13]

Out Cross Validation showing the lowest MAE and RMSE values.
The LightGBM Regression Model yielded better results for Happiness and Anxiety, while the Random Forest Regression Model was[14]

more eﬀective for Depression. In terms of future work, we should
focus on longer-term experiments to gather more data and minimize
overﬁtting risks, and explore other rPPG tools for better heart rate[15]

accuracy is also crucial. Finally, we hope this study will provide
valuable insights and methods for the further development of emotion monitoring and support systems, and expect to contribute to therealization of a healthier and happier society.

[16]

Acknowledgment
This work was supported by MEXT ”Innovation Platform for Society 5.0” Program Grant Number JPMXP0518071489.

[17]

Reference[1] Z.D. Cohen and R.J. DeRubeis, “Treatment selection in depression,”
Annual Review of Clinical Psychology, vol.14, no.1, pp.209–236,
2018. PMID: 29494258.
[2] M. Soleymani, S. Asghari-Esfeden, Y. Fu, and M. Pantic, “Analysis ofeeg signals and facial expressions for continuous emotion detection,”
IEEE Transactions on Aﬀective Computing, vol.7, no.1, pp.17–28,
2016.
[3] D. Dagar, A. Hudait, H.K. Tripathy, and M.N. Das, “Automatic emotion detection model from facial expression,” 2016 International Conference on Advanced Communication Control and Computing Technologies, pp.77–85, ICACCCT, 2016.
[4] M.A. Ozdemir, B. Elagoz, A. Alaybeyoglu, R. Sadighzadeh, and A.
Akan, “Real Time Emotion Recognition from Facial Expressions Using CNN Architecture,” 2019 Medical Technologies Congress, pp.1–
4, TIPTEKNO, 2019.
[5] A. Slimi, M. Hamroun, M. Zrigui, and H. Nicolas, “Emotion recog-

[18]

[19]

[20]

nition from speech using spectrograms and shallow neural networks,”
Proceedings of the 18th International Conference on Advances in
Mobile Computing & Multimedia, p.35–39, 2021.
A.S. Popova, A.G. Rassadin, and A.A. Ponomarenko, “Emotion
Recognition in Sound,” Advances in Neural Computation, Machine
Learning, and Cognitive Research, eds. by B. Kryzhanovsky, W.
Dunin-Barkowski, and V. Redko, pp.117–124, 2018.
L. Santamaria-Granados, M. Munoz-Organero, G. RamirezGonzález, E. Abdulhay, and N. Arunkumar, “Using deep convolutional neural network for emotion detection on a physiological signals
dataset (amigos),” IEEE Access, vol.7, pp.57–67, 2019.
S. Sriramprakash, V.D. Prasanna, and O.R. Murthy, “Stress detection
in working people,” Procedia Computer Science, vol.115, pp.359–
366, 2017. 7th International Conference on Advances in Computing
and Communications, ICACC-2017, 22-24 August 2017, Cochin, India.
R. Garg and S. Sengupta, “He is just like me: A study of the long-term
use of smart speakers by parents and children,” Proc. ACM Interact.
Mob. Wearable Ubiquitous Technol., vol.4, pp.1–24, mar 2020.
E. Smith, P. Sumner, C. Hedge, and G. Powell, “Smart speaker devices
can improve speech intelligibility in adults with intellectual disability,” International Journal of Language & Communication Disorders,vol.56, pp.583–593, 2021.
D. Fedotov, Y. Matsuda, and W. Minker, “From smart to personalenvironment: Integrating emotion recognition into smart houses,”2019 IEEE International Conference on Pervasive Computing andCommunications Workshops (PerCom ’19 Workshops), pp.943–948,
2019.
S. Kim and A. Choudhury, “Exploring older adults perception anduse of smart speaker-based voice assistants: A longitudinal study，”Computers in Human Behavior，vol.124，p.106914，2021．
A. Reis, D. Paulino, H. Paredes, and J. Barroso, “Using intelligent personal assistants to strengthen the elderlies’ social bonds,” Universal
Access in Human–Computer Interaction. Human and Technological
Environments, eds. by M. Antona and C. Stephanidis, pp.593–602,
2017.
F. Eyben, F. Weninger, F. Gross, and B. Schuller, “Recent developments in opensmile, the munich open-source multimedia feature
extractor,” Proceedings of the 21st ACM International Conference on
Multimedia, p.835–838, 2013.
F. Eyben, K.R. Scherer, B.W. Schuller, J. Sundberg, E. André, C.
Busso, L.Y. Devillers, J. Epps, P. Laukka, S.S. Narayanan, and K.P.
Truong, “The geneva minimalistic acoustic parameter set (gemaps)
for voice research and aﬀective computing,” IEEE Transactions on
Aﬀective Computing, vol.7, no.2, pp.190–202, 2016.
X. Niu, H. Han, S. Shan, and X. Chen, “Continuous heart rate measurement from face: A robust rppg approach with distribution learning,” 2017 IEEE International Joint Conference on Biometrics (IJCB),
pp.642–650, 2017.
Y.S. Dosso, A. Bekele, and J.R. Green, “Eulerian magniﬁcation ofmulti-modal rgb-d video for heart rate estimation,” 2018 IEEE International Symposium on Medical Measurements and Applications(MeMeA), pp.1–6, 2018.
A.R. M, Sahana, V. R, S. G, and S. N, “Heart rate detection througheulerian video magniﬁcation of face videos,” International Advanced
Research Journal in Science, Engineering and Technology, vol.10,
pp.486–492, 2023.
I. Fukui, “The depression and anxiety mood scale (dams): Scale development and validation,” Japanese Association of Behavioral andCognitive Therapies, vol.23, no.2, pp.83–93, 1997.
M. Hu, F. Qian, D. Guo, X. Wang, L. He, and F. Ren, “Eta-rppgnet:
Eﬀective time-domain attention network for remote heart rate measurement,” IEEE Transactions on Instrumentation and Measurement,
vol.70, pp.1–12, 2021.

—6—