# Model for Determining the Psycho-Emotional State of a Person Based on Multimodal Data Analysis
## 2 Related Works  
Over time, systems began to use fusion at different stages, such as early (at the function level) and late (weighted average of unimodal network results). In [12], a method for processing three types of multimodal data is proposed, which considers the delay and time shift of multimodal data in the time dimension and the relationship between embedding in multimodal data. The evaluation is based on the MOSEI dataset. Experimental results show that the method effectively improves the correlation of relevant information between multimodal signals. Combining features at the input level and combining deep neural networks for emotion analysis based on multimodal data is described in [15]. In [16; 17], the author first encodes each modality separately and then uses feature-level fusion in the middle layer to obtain a multimodal embedding, which is a static feature-level fusion. In contrast to the method described in previous articles, the author in [18; 19] uses dynamic fusion at the feature level, the essence of which is the simultaneous encoding of unimodal data and fusion at the feature level. Each information modality is given the same weight in the methods described above. However, this approach is only partially correct because during further research in articles [20; 21; 22; 23], researchers found that different modal information has other importance for the final result, and the degree of importance differs. For example, in [23], the author noted that verbal data would be more influential than non-verbal data, leading to a change in the word's meaning. The author takes the verbal modality as dominant and the non-verbal modality as an auxiliary for multimodal fusion. The papers [24; 25] analyze separated modalities for mood detection and modality and fusion. The multimodal-fusion approach in [25] uses a strategy referred to as \"late-fusion\" that involves the combination of unimodal model outputs as inputs of the decision engine. During the inception of this technology, there needed to be better quality of emotion determination. This problem has an explanation. Emotions reflect a person's subjective attitude to various phenomena. Emotion is a person's specific response to situations that affect his/her interests. There are emotional manifestations that cannot be controlled and that can be recognized. Under the influence of the same external factors, different people may have different emotions [26]. Emotions are manifested in different ways and in combination with each other. Recognizing emotions from facial expressions is a complex mental process. A person can recognize another person's emotions by their face, gait and gestures. [PERSON], a psychologist in the 1960s, believed that there were only six basic emotions expressed by facial emotions [27; 28]. But further research has shown much more variability regarding the number of emotional states and how they are expressed. This varies in different cultures [29], in different situations, and even within the same day. One of the companies offered police software to detect emotions [30]. The ability to detect emotions such as anger, stress or anxiety provides law enforcement with additional information when conducting a large-scale investigation. Ultimately, it is believed that the responsible use of this technology will be a factor that will make the world safer. Many job candidates have benefited from HireVue's technology, which aims to help eliminate the very significant human bias in the existing hiring process [31]. Cogito has developed voice analysis algorithms for call center personnel to help them detect when customers are experiencing problems. However, people believe that before emotion detection can make automated decisions, the industry needs more evidence that machines can detect human emotions effectively and consistently [32]. The field of recognition of a person's psycho-emotional state by a machine gained popularity with the advent of machine learning and neural networks.