# Multimodal Emotion Recognition Using Visual, Vocal and Physiological Signals: A Review
## 4 Human Emotional Expression Recognition
### Multimodal Emotion Recognition
In contrast, visual data convey facial expressions, which can further aid in understanding the user's affective state. Moreover, a combination of video data and physiological data should enhance the recognition of the human affective state and create a better emotion model. Multimodal emotion computing integrates the outputs of unimodal systems by fusing/combining information from different modalities for analysis using various (multimodal) fusion techniques for data collected from multimodal sources. Existing techniques for fusing information from different modalities for emotion computing are categorized into two groups: decision-based fusion and feature-based fusion. Data are processed as a combined entity in feature-level fusion, or data are split into modalities and then analyzed in decision-level fusion [47]. Some algorithms use a combination of both of these approaches or rely on specified rules to fuse modalities. The framework of a typical multimodal emotion recognition system is given in Figure 7. It consists of two fundamental steps: processing unimodal data separately and fusing them all. Both steps are equally important. The multimodal system proceeds via the fusion of unimodal information. Unimodal emotion recognition, an essential preprocessing step for multimodal fusion, must perform well to build an intelligent multimodal system. Poor analysis of a modality can worsen the multimodal system's performance, while an inefficient fusion method can ruin the multimodal system's stability. Researchers can identify essential and appropriate visual, audio, and physiological machine learning analysis methods from the literature and fuse them using state-of-the-art methods. Multimodal data, such as the data obtained from a video, can be a valuable source of information for emotion analysis, but major challenges need to be addressed. For example, the ways we convey and express opinions vary from one person to another. Some people may express their emotions more vocally, while others do so more visually. When people express their emotions with more vocal modulation, the audio data may contain major cues for emotion mining. On the other hand, when a person is using more facial expressions, most of the cues needed for emotion mining can be assumed to reside in their facial expressions. Single modalities exhibit limitations in terms of meeting robustness, accuracy, and overall performance requirements, which greatly restrict such systems' usefulness in practical, real-world applications. The key advantages associated with multimodal approaches are that they are likely to generate more realistic results and provide the possibility of realistic human-to-machine interactions in real-time [47]. The aim of data fusion (i.e., multimodal recognition approach) is to increase the accuracy and reliability of estimates. However, multimodal analysis and processing are complex and still in the development stage. Before real-time multimodal emotion recognition makes its way into our everyday lives (for example, enabled on smart devices), some of the major research challenges that need to be addressed to effectively integrate various modalities include synchronization that is hindered by noisy characteristics in different sensors (audio and speech in a video). Synchronization involves integrating information from different sources across different time scales and measurement values. The lack of data synchronization from heterogeneous inputs hinders the modeling of temporal information. Multimodal approaches also require more data, given the increase in input dimension (quantity of data needed in multimodal approaches increases as the input dimension increases), which raises the issues of the limitation in data available for analysis and handling the vast quantities of data involved. For real-time analysis of multimodal big data, an appropriately scalable big data architecture and platform must be designed to effectively cope with the heterogeneous big data challenges of growing space and time complexity.