# DeepVANet: A Deep End-to-End Network for Multi-modal Emotion Recognition

Yuhao Zhang, Md Zakir Hossain, Shafin Rahman

► To cite this version:

Yuhao Zhang, Md Zakir Hossain, Shafin Rahman. DeepVANet: A Deep End-to-End Network for Multi-modal Emotion Recognition. 18th IFIP Conference on Human-Computer Interaction (INTER-ACT), Aug 2021, Bari, Italy. pp.227-237, 10.1007/978-3-030-85613-7_16 . hal-04292350

HAL Id: hal-04292350

https://inria.hal.science/hal-04292350v1

Submitted on 17 Nov 2023

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# DeepVANet: A Deep End-to-End Network for Multi-modal Emotion Recognition

Yuhao Zhang¹, Md Zakir Hossain¹,²[0000-0003-1892-831X], and
Shafin Rahman³[0000-0001-7169-0318]

¹ The Australian National University, Canberra, ACT 2601, Australia

² CSIRO Agriculture and Food, Black Mountain, Canberra, ACT 2601, Australia

³ North South University, Dhaka, Bangladesh

`{yuhao.zhang,zakir.hossain}@anu.edu.au, shafin.rahman@northsouth.edu`

**Abstract.** Human facial expressions and bio-signals (e.g., electroencephalogram and electrocardiogram) play a vital role in emotion recognition. Recent approaches employ both vision-based and bio-sensing data to design multi-modal recognition systems. However, these approaches require tremendous domain-specific knowledge, complex pre-processing steps and fail to take full advantage of the end-to-end nature of deep learning techniques. This paper proposes a deep end-to-end framework, DeepVANet, for multi-modal valence-arousal-based emotion recognition that applies deep learning methods to extract face appearance features and bio-sensing features. We use convolutional long short-term memory (ConvLSTM) techniques in face appearance feature extraction to capture spatial and temporal information from face image sequences. Unlike conventional time or frequency domain features (e.g., spectral power and average signal intensity), we use a 1D convolutional neural network (Conv1D) to learn bio-sensing features automatically. In experiments, we evaluate our method using DEAP and MAHNOB-HCI datasets. Our proposed multi-modal framework successfully outperforms both single- and multi-modal methods achieving superior performance compared to state-of-the-art approaches and reaches as high as 99.22% correctness.

**Keywords:** Emotion recognition · Deep learning · Physiological signals

## 1 Introduction

Emotion is an important mental state of human beings and dominates people's attitudes and behaviors. Recently, emotion recognition has become a popular research topic due to its wide applications in the medical, education, and gaming fields [5,35]. In particular, existing works predominantly investigate two modalities of data for the emotion recognition task. *Firstly*, vision-based data, i.e., face image/video, widely dominates emotion recognition research. Facial expressions can reflect human's emotions directly, and they are easy to capture by cameras. State-of-the-art studies have successfully explored both hand-crafted and deep learning methods to extract these kinds of facial expression features [16]. Hand-crafted methods include facial specific methods based on facial landmarks (angle
and distance of landmark pairs) [8,37] and other effective general visual feature
extraction methods such as Local Binary Pattern (LBP) [10] and Histogram
of Grandients (HoG) [20,34]. Deep learning methods, particularly convolutional
neural network (CNN) and its variations, have shown their powerful visual data
processing ability in facial emotion recognition tasks [21]. Especially for facial
image sequences or videos, 3D CNN [9], recurrent convolutional neural network
(RCNN) [14,23] and ConvLSTM [11] perform well in spatial and temporal fa-
cial expression feature extraction. Secondly, physiological/bio-sensing signals,
i.e., electroencephalogram (EEG), electrocardiogram (ECG), and galvanic skin
response (GSR) data are used to extract discriminative information of human
emotions [27,31]. Compared to facial expressions, physiological signals are more
instinctive and uncontrolled. So, they can reflect the real emotions more ac-
curately. Methods use hand-crafted features like power spectral density (PSD)
from different frequency bands, conditional cross-entropy, and statistical features
to extract discriminative information [18]. However, due to inconsistency across
hardware, the necessity of expert pre-processing, and the expensive nature of
bio-sensing data, there is not enough work on deep learning strategies.

Apart from single modality (either bio-sensing or vision) approaches, some multi-modal affective computing efforts have been proposed and shown advantages in emotion recognition accuracy [25,28,38]. Single modal features have their respective strength in some aspects. Thus, their fusion can take advantage of their strengths by generating comprehensive and salient multi-modal features. While dealing with multi-modal data, current literature does not utilize the full benefit of deep learning (e.g. complicated data pre-processing). This paper investigates an end-to-end approach to address the emotion recognition task using both modalities. Human emotional states can be represented qualitatively (discrete model) [6] or quantitatively (dimensional model) [24]. Discrete models generally classifies emotions into six basic types: happiness, sadness, anger, surprise, disgust, and fear. The dimensional model represents emotions in a multidimensional space, such as valence-arousal space, which is more flexible and represents a broader range of emotions. This study focuses on emotion recognition in valence-arousal space using multi-modal data (vision-based and bio-sensing).

**Limitation of Prior Works:** We identify several issues in existing works of the valence-arousal model of emotion classification: (a) Although some models used deep learning in vision pipeline, they failed to incorporate the same on bio-sensing data [12]. (b) Some models pre-processed raw bio-sensing signals to 2D-frames (e.g. EEG heatmap of PSD bands) to extract CNN features [13,28,36]. (c) Other models usually extracted individual modal features separately and then applied a decision-level fusion on them [12,25]. All the mentioned issues hamper the end-to-end nature of deep learning.

This paper proposes a deep end-to-end network, DeepVANet, to recognize humans emotions using both face videos and physiological signals by extracting vision-based and bio-sensing features, respectively. At the vision pipeline, we use a Convolutional Recurrent Neural Network (CRNN) based on convolutional long short-term memory (ConvLSTM) as face appearance feature extractor to obtain
spatio-temporal information from face videos [26]. Similarly, we use Conv1D neural network at the bio-sensing pipeline instead of traditional hand-crafted time/frequency-based extraction methods. Later, we employ a fusion pipeline to fuse both modality features enriching the feature quality for better classification. Our fusion pipeline supports both feature- and decision-level fusion strategies. Different modal features are concatenated together then pass through some fully-connected (FC) classifier layers to predict the valence/arousal label for feature-level fusion. In contrast, for decision-level fusion, FC sub-classifiers accept single modal features and output predict scores separately. Then, adaptive boosting techniques are employed to yield the final emotion label. Our method combines all steps while maintaining the end-to-end nature of the work. We present experimental results on DEAP [17] and MAHNOB-HCI [29] datasets demonstrating new state-of-the-art results on multi-modal emotion recognition. In summary, the main contributions of our work are: (1) To the best of our knowledge, we propose the first end-to-end deep network for multi-modal (face and bio-sensing) valence-arousal-based emotion recognition. (2) Instead of the traditional use of hand-crafted features, we fuse Conv1D based bio-sensing and CRNN/ConvLSTM based vision feature to learn a rich feature representation jointly for emotion classification. (3) We perform extensive experiments on two well-known datasets achieving state-of-the-art performance: 98.56/98.18% and 99.19/99.22% (valence/arousal) on DEAP and MAHNOB-HCI, respectively.

# 2 Multi-modal Emotion Recognition

**Problem Formulation:** Suppose, for ith facial emotion video frames, $F_i = \langle I_t|1...n_i \rangle$ where, $I_t$ represents tth image frame, we collect bio-sensing signals, $B_i = \langle e_t|1...m_i \rangle$, where, $e_t$ represent tth data point. $n_i$ and $m_i$ are the length of the video and physiological signal, respectively. For ith instance the ground-truth annotation is $y_i$ which can be valence or arousal value. We train an end-to-end parameterized model, $\mathcal{F}_{\theta}$ using a set of tuples $\{(\mathbf{F}_i, \mathbf{B}_i), \mathbf{y}_i\} : i \in [0, \mathcal{T}]$, $\mathbf{y}_i \in [0, 1]$, where $\mathcal{T}$ is the total number of instances in the dataset, $\mathbf{y}_i = 0$ represents low valence/arousal, high valence/arousal otherwise. During inference, given a test video and physiological signal pair ($\mathbf{F}_j, \mathbf{B}_j$) as input, $\mathcal{F}_{\theta}$ predicts $\hat{\mathbf{y}}_j$ which approximates ground-truth annotation $\mathbf{y}_j$ as follows: $\hat{\mathbf{y}}_j = \mathcal{F}_{\theta}((\mathbf{F}_j, \mathbf{B}_j); \Theta)$

## 2.1 Architecture

Our proposed DeepVANet for VA based emotion recognition has three blocks: Face Appearance, Bio-sensing Data, and Classification Pipelines (Fig. 1).

**Face Appearance Pipeline:** We deploy 3-layer CNN (pretrained on AFEW-VA dataset [19]) at the top of our face appearance feature extractor to capture high-level spatial information. The inputs of the face appearance feature extractor are face image sequences (with size of 64 × 64). Each sample $F_i$ has the shape of $n \times 3 \times 64 \times 64$, where n denotes the sequence length and 3 represents the number of RGB channels. The kernel number of three convolutional
<figure><img src="image_3.png" /><figcaption>**Fig. 1.** Our proposed DeepVANet for Valance and Arousal based emotion recognition; FC and BN stand for fully connected layer and batch normalisation respectively.</figcaption></figure>

layers are 32, 64, and 128, respectively, where all the kernels have the same size of 3×3. Each layer is followed by a ReLu activator and a 3×3 max pooling. So, through a series of 2D convolutional and max-pooling operations, the CNN layers output the feature maps $F_t'$ with the shape of $n \times 128 \times 6 \times 6$. Facial expressions are dynamic and evolve during the video containing temporal patterns. We extract face expression information in both spatial and temporal dimensions using a ConvLSTM [26]. For each timestep $t$, input gate $i_t$, forget gate $f_t$, cell gate $g_t$ and output gate $o_t$ are defined as $i_t = \sigma(W_{xi} * F_t' + W_{hi} * h_{t-1} + b_i)$, $f_t = \sigma(W_{xf} * F_t' + W_{hf} * h_{t-1} + b_f)$, $g_t = \tanh(W_{xg} * F_t' + W_{hg} * h_{t-1} + b_g)$ and $o_t = \sigma(W_{xo} * F_t' + W_{ho} * h_{t-1} + b_o)$; cell state $c_t$ and hidden state $h_t$ are defined as $c_t = f_t \odot c_{t-1} + i_t \odot g_t$ and $h_t = o_t \odot \tanh(c_t)$, where $\odot$ and $*$ demote the Hadamard product and 2D convolutional operator. A followed FC layer accepts the final hidden state and outputs the face appearance features which is a $1 \times \lambda_1$ vector $f_{face} = (a_1, a_2, ..., a_{\lambda_1})$.

**Bio-sensing Data Pipeline:** Instead of traditional time or frequency domain methods, we use deep learning methods to extract bio-sensing features automatically. To extract high-level bio-sensing features, we apply a multi-layer CNN architecture consisting of three 1D convolutional layers and a fully connected layer [32]. Each convolutional layer is followed by a batch normalization layer and a ReLu activator, and the last fully connected layer is deployed for further feature extraction and flattening. The inputs to the bio-sensing data pipeline are multi-channel physiological signals (concatenate EEG and peripheral physiological signals over channel dimension). For each bio-sensing input instance $B_i$ with the shape of $c_{bio} \times m$, where $c_{bio}$ and $m$ represent the channel number and signal length respectively, the forward propagation in our bio feature extractor is shown as follows. Firstly, $B_i$ flows into three-layer 1D CNN. The kernel sizes
of three 1D CNN layers are 5, 3, 3, respectively. We choose to decrease output channel numbers for CNN layers 24, 16, and 8, to reduce the bio-sensing feature size and computation complexity. So, the output of convolutional layers is in the shape of 8 × (m − 8), where m denotes the signal length. Then it is put into a FC layer and flattened into a 1 × λ₂ bio feature vector fbio = (b₁, b₂, ..., bλ₂).

**Classification Pipeline:** The classification pipeline predicts a score for valence or arousal. For each instance, we have ground-truth ŷ ∈ {0, 1} to represent the emotion label (valence or arousal), where 0 and 1 represent low and high, respectively. Here, we investigate two kinds of fusion methods (feature-level and decision-level fusion) to make full use of different modalities. For feature-level fusion, face appearance feature fface and bio feature fbio are concatenated to generate a multi-modal feature vector fmulti = fface ⊕ fbio, which is fed into the fusion classifier containing a two-layer fully-connected neural network. Using different value for λ₁ and λ₂, we can set different feature size combinations and explore the contribution of each single modality. For decision-level fusion, we use Adaptive Boosting (AdaBoost) [7,12]. The face and bio-sensing features are used to train two sub-classifiers separately. The sub-classifier network is the same as the fusion classifier. Each sub-classifier yields a predicted score Si, which are used together to calculate a fusion score Sfusion using the Adaboost to get the proper weights of each sub-classifier. Generally, the sub-classifier having the less predict error holds the higher weight in calculating the fusion score. Training samples are also given the weights which influence their contribution to the error. The sample weights are initialized uniformly. The weight decreases if the sample is predicted correctly and increases otherwise. After finishing all training iterations, we can get the sub-classifier weights and calculate the fusion score.

## 2.2 Training and Inference

We use binary cross-entropy as a loss function for both feature- and decision-level fusion cases. Considering N, y and Si as the training batch size, target emotion label and predicted score, respectively, the loss is calculated as:

$$Loss = -\frac{1}{N} \sum_{i=1}^{N} (y_i \cdot \ln S_i + (1 - y_i) \cdot \ln (1 - S_i))$$

During inference, we forward test video, Fj and physiological signal, Bj to our proposed network, obtain score Sfusion and use the following formula for final prediction, ŷ.

$$ŷ = \begin{cases} High, & S_{fusion} > 0.5 \\ Low, & S_{fusion} \le 0.5 \end{cases}$$

# 3 Experiment

**Dataset:** Two public multi-modal emotion datasets, namely DEAP, MAHNOB-HCI are used to evaluate our network separately. A brief statistics about the dataset is shown in Table 1.
<table><caption>**Table 1.** Statistics of the datasets used in this work</caption><thead><tr><th>Criterion</th><th>DEAP</th><th>MAHNOB-HCI</th></tr></thead><tbody><tr><td>Dataset size</td><td>22 subjects × 40 trials</td><td>27 subjects × 20 trials</td></tr><tr><td>Stimulus duration</td><td>60 s</td><td>49.7-117 s</td></tr><tr><td>Modality</td><td>Vision, EEG, Peripheral EEG, EOG, EMG, GSR,</td><td>Vision, EEG, Peripheral EEG, ECG, GSR, Respiration</td></tr><tr><td>Used bio signals</td><td>Respiration belt, Plethysmograph, Temperature</td><td>belt, Temperature</td></tr><tr><td>Labels</td><td>Continuous valence and arousal from 1 to 9</td><td>Discrete valence and arousal of integers from 1 to 9</td></tr></tbody></table>

**Experimental Evaluation:** In line with the previous works [12,28], we divide valence and arousal into ‘High’/‘Low’ using value = 5 as the threshold. We notice that physiological signals vary significantly across subjects [2,17,36], and this inter-subject difference affects the emotion recognition accuracy while predicting valence and arousal across people. Therefore, similar to the past works [2,36], we train and evaluate our model for every single subject (called per-subject experiment) to improve the recognition performance. In addition to the per-subject experiment, we also conduct inter-subject experiments. Here, all subjects’ data is used to train and test one model aiming to evaluate our proposed network’s generalization ability. For both per- and inter-subject experiments, we run 10-fold cross-validation and take the average testing accuracy as the model performance metric. Besides our multi-modal network, we also evaluate two single modalities, face and bio-sensing, separately. Furthermore, EEG and peripheral physiological signals are used separately in bio-sensing modality experiments.

**Validation Strategy:** We have two hyper-parameters in our model: the face feature dimension $\lambda_1$ and the bio-sensing feature dimension $\lambda_2$. For both hyper-parameter cases, we choose a set of candidates {16, 32, 64, 128, 256}, and perform a grid search to determine the best hyper-parameter. We use the mean recognition accuracy of valence and arousal as validation metric, and 16 and 64 are chosen as the face appearance feature and bio-sensing feature size.

**Implementation Details**^4: To overcome the limited data size for deep learning, we divide each trial into 1-second length segments as proposed in [36]. We extract frames from face videos at 5 Hz for vision-based data and perform face detection [3], cropping, and alignment on each frame. These face images are resized to 64×64 resolution and normalized within the interval [0, 1] before feeding into our model. For bio-sensing data, we perform average reference, 4-45 Hz bandpass filter, and artifact removal using EEGLab [4] on 32-channel EEG signals. Both EEG and peripheral signals are downsampled to 128 Hz and concatenated together along with channel dimension. For each segment, we apply the baseline removal on bio-sensing signals by subtracting the mean of 3-second pre-trial data [36]. We use the Adam algorithm [15] as the optimizer during training. The batch size, numbers of training epochs, and learning rate for all

<footnote>^4 Code and evaluation are available at: https://github.com/geekdanielz/DeepVANet</footnote>
<table><caption>**Table 2.** Overall results of VA based emotion recognition. Both inter-subject and per-subject experiment results are reported. For accuracy (%), valence at left and arousal at right.</caption><thead><tr><th>Data</th><th>Study</th><th>Modalities</th><th>Accuracy</th></tr></thead><tbody><tr><td rowspan="10">DEAP</td><td colspan="3">Inter-subject experiments</td></tr><tr><td>Siddharth et al. [28]</td><td>Face, EEG, Peripheral</td><td>79.52/78.34</td></tr><tr><td>Tang et. al [30]</td><td>EEG, Peripheral</td><td>83.82/83.23</td></tr><tr><td><strong>This work</strong></td><td>Face, EEG, Peripheral</td><td><strong>95.56/95.33</strong></td></tr><tr><td colspan="3">Per-subject experiments</td></tr><tr><td>Keoltra et al. [17]</td><td>EEG, Peripheral</td><td>62.70/62.00</td></tr><tr><td>Huang et al. [12]</td><td>Face, EEG</td><td>80.30/74.23</td></tr><tr><td>Liu et al. [22]</td><td>EEG, Peripheral</td><td>85.20/80.50</td></tr><tr><td>Alhagry et. al [1]</td><td>EEG</td><td>85.65/85.45</td></tr><tr><td>Yang et al. [36]</td><td>EEG</td><td>90.80/91.03</td></tr><tr><td>Anubhav et al. [2]</td><td>EEG</td><td>94.69/93.13</td></tr><tr><td><strong>This work</strong></td><td>Face, EEG, Peripheral</td><td><strong>98.56/98.18</strong></td></tr><tr><td rowspan="10">MAHNOB</td><td colspan="3">Inter-subject experiments</td></tr><tr><td>Soleymani et al. [29]</td><td>EEG, Peripheral</td><td>57.00/52.40</td></tr><tr><td>Wiem and Lachiri [33]</td><td>EEG, Peripheral</td><td>68.75/64.23</td></tr><tr><td>Siddharth et al. [28]</td><td>Face, EEG, Peripheral</td><td>85.49/82.93</td></tr><tr><td><strong>This work</strong></td><td>Face, EEG, Peripheral</td><td><strong>96.82/97.79</strong></td></tr><tr><td colspan="3">Per-subject experiments</td></tr><tr><td>Keoltra et al. [18]</td><td>Face, EEG</td><td>74.00/70.00</td></tr><tr><td>Huang et al. [12]</td><td>Face, EEG</td><td>75.21/75.63</td></tr><tr><td><strong>This work</strong></td><td>Face, EEG, Peripheral</td><td><strong>99.19/99.22</strong></td></tr></tbody></table>

<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Ablation study results (recognition accuracy %) for DEAP dataset</figcaption></figure>

face-, bio-, and multi-modal are 64, 50, and $10^{-3}$, respectively. We implement our work with the Pytorch framework using a single Nvidia V100 GPU.

## 3.1 Quantitative Results

**Overall Result:** Table 2 reports the inter- and per-subject experiment results of proposed DeepVANet on DEAP and MAHNOB-HCI datasets. We choose [17,29] as baseline methods and consider some studies applying deep learning for comparison. These studies used the same binary VA labels as this paper. Some notable observations are: (1) Multi-modal methods [12,18,28] achieved better accuracy than single model methods [17,29] which justifies the significance of fusion strategy on face images and physiological signals. (2) Some deep learning-based approaches [1,2,36] get an accuracy above 90% by only using EEG signals. This is because the deep learning-based (LSTM) EEG features are more discriminative than handcrafted features. (3) We notice that the per-subject experiment result is better than the inter-subject experiment because the bio-sensing data variance across subjects impacts emotion recognition performance. However, our DeepVANet also performs well in inter-subject experiments and shows a good generalization ability to predict emotion for different people. (4) Overall, our feature-level fusion approach achieves the highest average recognition accuracy of 98.56%/98.18% and 99.19%/99.22% on DEAP and MAHNOB-HCI datasets, respectively, and surpasses the previous methods by a considerable margin. This
<figure><img src="image_5.png" /><figcaption>**Fig. 3.** 2D t-SNE visualization of features extracted from subject 1 of DEAP dataset; top and bottom rows represent valence and arousal, respectively.</figcaption></figure>

success is a result of utilizing advanced deep learning-based feature extraction for bio-sensing data.

**Ablation Study:** To explore the emotion recognition performance of different modalities, we test our approach using both single- and multi-modal input on DEAP dataset. From the results in Fig 2, we find that: (1) EEG features are more salient than peripheral features, and bio-sensing modality (EEG+Peripheral) performs better than individual EEG and peripheral modality. (2) multi-modality outperforms all the single modalities, which demonstrates the effectiveness of our feature- and decision-level fusion strategies. (3) feature-level fusion method performs better than decision-level fusion in most cases. We believe it is because we only have two sub-classifiers in our approach, limiting the effectiveness of the decision-level fusion based on AdaBoost.

## 3.2 Qualitative Visualization

We use t-SNE to reduce the feature dimension into 2D space and visualize different modal features in Fig. 3. We find that the HIGH and the LOW class features successfully form two rough clusters in 2D space. Notably, for the fusion modal, two clusters are separated more clearly. It also demonstrates that our multi-modal features are salient and discriminative.

## 4 Conclusion

In this paper, we propose a deep end-to-end network, DeepVANet, for multi-modal emotion recognition. The network accepts face image sequences and bio-sensing signals (e.g., EEG, EOG, ECG, GSR etc.) as input and yields valence-arousal labels for emotion recognition. We adopt ConvLSTM and Conv1D to extract face appearance and bio-sensing computing features, respectively, demonstrating a significant advantage in the valence-arousal emotion recognition model. Furthermore, we evaluate our method with per-subject and inter-subjects experiments. For both strategies, the experiments' results show that our end-to-end network outperforms previous studies and establishes new state-of-the-art performance on DEAP and MAHNOB-HCI datasets.
References

1. Alhagry, S., Fahmy, A.A., El-Khoribi, R.A.: Emotion recognition based on eeg using lstm recurrent neural network. Emotion 8(10), 355–358 (2017)

2. Anubhav, Nath, D., Singh, M., Sethia, D., Kalra, D., Indu, S.: An efficient approach to eeg-based emotion recognition using lstm network. In: IEEE International Colloquium on Signal Processing & Its Applications (CSPA). pp. 88–92 (2020)

3. Bulat, A., Tzimiropoulos, G.: How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks). In: International Conference on Computer Vision (2017)

4. Delorme, A., Makeig, S.: Eeglab: an open source toolbox for analysis of single-trial eeg dynamics including independent component analysis. Journal of neuroscience methods 134(1), 9–21 (2004)

5. Dzedzickis, A., Kaklauskas, A., Bucinskas, V.: Human emotion recognition: Review of sensors and methods. Sensors 20(3), 592 (2020)

6. Ekman, P.: Are there basic emotions? Psychological review pp. 550–553 (1992)

7. Freund, Y., Schapire, R.E., et al.: Experiments with a new boosting algorithm. In: icml. vol. 96, pp. 148–156 (1996)

8. Ghimire, D., Lee, J.: Geometric feature-based facial expression recognition in image sequences using multi-class adaboost and support vector machines. Sensors 13(6), 7714–7734 (2013)

9. Haddad, J., Lézoray, O., Hamel, P.: 3d-cnn for facial emotion recognition in videos. In: International Symposium on Visual Computing. pp. 298–309. Springer (2020)

10. Happy, S., George, A., Routray, A.: A real time facial expression classification system using local binary patterns. In: 2012 4th International conference on intelligent human computer interaction (IHCI). pp. 1–5. IEEE (2012)

11. Huang, J., Li, Y., Tao, J., Lian, Z., Yi, J.: End-to-end continuous emotion recognition from video using 3d convlstm networks. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 6837–6841 (2018)

12. Huang, Y., Yang, J., Liu, S., Pan, J.: Combining facial expressions and electroencephalography to enhance emotion recognition. Future internet 11(5), 105 (2019)

13. Jinliang, G., Fang, F., Wang, W., Ren, F.: Eeg emotion recognition based on granger causality and capsnet neural network. In: International Conference on Cloud Computing and Intelligence Systems (CCIS). pp. 47–52 (2018)

14. Kahou, S.E., Michalski, V., Konda, K., Memisevic, R., Pal, C.: Recurrent neural networks for emotion recognition in video. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction. pp. 467–474 (2015)

15. Kingma, D.P., Ba, J.L.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

16. Ko, B.C.: A brief review of facial emotion recognition based on visual information. sensors 18(2), 401 (2018)

17. Koelstra, S., Muhl, C., Soleymani, M., Lee, J.S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., Patras, L.: Deap: A database for emotion analysis using physiological signals. IEEE Transactions on Affective Computing 3(1), 18–31 (2012)

18. Koelstra, S., Patras, I.: Fusion of facial expressions and eeg for implicit affective tagging. Image and Vision Computing 31(2), 164–174 (2013)

19. Kossaifi, J., Tzimiropoulos, G., Todorovic, S., Pantic, M.: Afew-va database for valence and arousal estimation in-the-wild. Image and Vision Computing 65 (2017)

20. Kumar, P., Happy, S., Routray, A.: A real-time robust facial expression recognition system using hog features. In: 2016 International Conference on Computing, Analytics and Security Trends (CAST). pp. 289–293. IEEE (2016)
21. Lee, J., Kim, S., Kim, S., Park, J., Sohn, K.: Context-aware emotion recognition networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10143–10152 (2019)

22. Liu, W., Zheng, W.L., Lu, B.L.: Emotion recognition using multimodal deep learning. In: International conference on neural information processing. Springer (2016)

23. Nie, W., Ren, M., Nie, J., Zhao, S.: C-gcn: Correlation based graph convolutional network for audio-video emotion recognition. IEEE Transactions on Multimedia pp. 1–1 (2020)

24. Panoulas, K.J., Hadjileontiadis, L.J., Panas, S.M.: Brain-computer interface (bci): types, processing perspectives and applications. In: Multimedia Services in Intelligent Environments, pp. 299–321. Springer (2010)

25. Salama, E.S., El-Khoribi, R.A., Shoman, M.E., Shalaby, M.A.W.: A 3d-convolutional neural network framework with ensemble learning techniques for multi-modal emotion recognition. Egyptian Informatics Journal (2020)

26. Shi, X., Chen, Z., Wang, H., Yeung, D.Y., kin Wong, W., chun Woo, W.: Convolutional lstm network: A machine learning approach for precipitation nowcasting. Advances in Neural Information Processing Systems 28, 802–810 (2015)

27. Shu, L., Xie, J., Yang, M., Li, Z., Li, Z., Liao, D., Xu, X., Yang, X.: A review of emotion recognition using physiological signals. Sensors 18(7), 2074 (2018)

28. Siddharth, Jung, T.P., Sejnowski, T.J.: Utilizing deep learning towards multi-modal bio-sensing and vision-based affective computing. IEEE Transactions on Affective Computing (2019)

29. Soleymani, M., Lichtenauer, J., Pun, T., Pantic, M.: A multimodal database for affect recognition and implicit tagging. IEEE Transactions on Affective Computing 3(1), 42–55 (2012)

30. Tang, H., Liu, W., Zheng, W.L., Lu, B.L.: Multimodal emotion recognition using deep neural networks. In: International Conference on Neural Information Processing. pp. 811–819. Springer (2017)

31. Torres, E.P., Torres, E.A., Hernandez-Alvarez, M., Yoo, S.G.: Eeg-based bci emotion recognition: A survey. Sensors 20(18), 5083 (2020)

32. Ullah, I., Hussain, M., Aboalsamh, H., et al.: An automated system for epilepsy detection using eeg brain signals based on deep learning approach. Expert Systems with Applications 107, 61–71 (2018)

33. Wiem, M.B.H., Lachiri, Z.: Emotion classification in arousal valence model using mahnob-hci database. International Journal of Advanced Computer Science and Applications 8(3) (2017)

34. Yan, J., Zheng, W., Xu, Q., Lu, G., Li, H., Wang, B.: Sparse kernel reduced-rank regression for bimodal emotion recognition from facial expression and speech. IEEE Transactions on Multimedia 18(7), 1319–1329 (2016)

35. Yang, Y., Hossain, M.Z., Gedeon, T., Rahman, S.: Realsmilenet: A deep end-to-end network for spontaneous and posed smile recognition. In: Proceedings of the Asian Conference on Computer Vision (ACCV) (November 2020)

36. Yang, Y., Wu, Q., Qiu, M., Wang, Y., Chen, X.: Emotion recognition from multi-channel eeg through parallel convolutional recurrent neural network. In: 2018 International Joint Conference on Neural Networks (IJCNN). pp. 1–7. IEEE (2018)

37. Zeng, Z., Tu, J., Liu, M., Huang, T.S., Pianfetti, B., Roth, D., Levinson, S.: Audio-visual affect recognition. IEEE Transactions on multimedia 9(2), 424–428 (2007)

38. Zhang, H.: Expression-eeg based collaborative multimodal emotion recognition using deep autoencoder. IEEE Access 8, 164130–164143 (2020)