# Multimodal Emotion Recognition Using Visual, Vocal and Physiological Signals: A Review
In light of these considerations, this article reviews subtle emotion recognition and contrasts such recognition with macro-expression recognition. This article also reviews multimodal emotion recognitions that consider spontaneous visual or vocal expressions or involuntary physiological arousal. These modalities exhibit an instantaneous nature and play a critical role in various fields, including security, healthcare (such as patient care and therapeutic support), and human-computer interactions, particularly in systems involving robots capable of perceiving and responding to human emotions. The main contributions of this study are listed as follows: we examine the state of the art in multimodal emotion recognition based on spontaneous emotion modalities and subtle emotion recognition to reveal the emotional state related to the true feeling of an individual. This paper also discusses the use of deep learning to improve emotion recognition and its current challenges. Subtle expressions are solely identified through dynamic displays/expressions such as dynamic facial expressions and dynamic emotional body movements that effectively emulate the 3D spatio-temporal processing style associated with a human brain when recognizing emotions. Given that the human brain activity associated with the interpretation of emotion has spatio-temporal dynamics, this review article discusses the state-of-the-art methodologies in dynamic expression. The challenges of deep learning for effective automatic emotion recognition are highlighted. Numerous research review papers have recently discussed multimodal emotion recognition, highlighting its strengths and limitations [36; 37; 38; 39; 40; 41; 42; 43; 44; 45; 46]. Existing reviews often focus on multimodal emotion recognition in the context of audio-visual integration and, to a lesser extent, incorporate electroencephalogram (EEG) data while overlooking other physiological data such as heart rate and skin conductance. They do not look at the aspect of subtle expressions and dynamic expressions. They also require more scrutiny regarding the robustness and generalizability of methods, particularly with respect to bias. Although they present fusion methods, they must investigate or compare them across varied contexts to identify the best practices. Therefore, we address these gaps and examine, in large part, the methodological characteristics unique to multimodal emotion recognition, combining visual, vocal or physiological signals. The research question for this study is as follows:  
* RQ1: How can multimodal emotion recognition methods, using visual, vocal and physiological signals be optimized to enhance robustness and accuracy, and what is the impact of deep learning techniques and dynamic expression analysis in overcoming these challenges?In this study, we identify the limitations of current approaches to multimodal emotion recognition. We focus on the fine-grained recognition of emotional states and present emotion recognition in a way that enhances the explainability of a classifier's output, thereby ensuring better user adoption and trust in emotional computing. We categorize multimodal and subtle emotion recognition techniques into distinct groups based on their implementations, providing concise explanations for each and comparing their performance. We conduct a comprehensive analysis of the current state of emotion recognition, similar to ref. [47], by evaluating the performance of various methods on the same dataset. The remaining sections of this paper are organized as follows. Section 2 discusses the research methods employed in conducting this review study. In Section 3, the human emotion categorizations are reviewed. In Section 4, automatic human emotion recognition methods are elaborated on and reviewed. Then, an overview of multimodal emotion analysis research is provided. Deep learning solutions and challenges are introduced in Section 5. Section 6 discusses the findings and current limitations of multimodal emotion recognition.