# Model for Determining the Psycho-Emotional State of a Person Based on Multimodal Data Analysis
## 1 Introduction  
More than a decade ago, the realm of human-machine interaction began gaining traction, progressing toward increasingly user-friendly and adaptable machine interfaces. With the emergence of machine learning, the capability to discern human emotions became feasible [1]. Examining interpersonal interaction underscores the importance of comprehending the psycho-emotional state of conversational partners. Consequently, the integration of emotion recognition into machine-human interaction processes has surged in popularity [2]. Traditionally, humans perceived machines as emotionless entities primarily executing intricate computations. However, equipping machines with the ability to decipher facial expressions, intonation, gestures, and other non-verbal cues promises enhanced rapport between machines and individuals [3]. Many global car brands often introduce systems that warn the driver about his drowsiness [4] or feeling unwell. Popular brands are also willing to pay money to evaluate human emotions while viewing advertisements [5]. A human behavior recognition system is often helpful in predicting crimes [6]. And, there are many other areas in which understanding the emotions of consumers would play the role of an essential push in development [7]. Some reports suggest that the emotion detection and recognition market was worth about $12 billion in 2018 and could grow to more than $90 billion by 2024 [8]. The Israeli startup Beyond Verbal (Tel Aviv-Yafo, Israel) [9] is developing intelligent emotion analytics that look for signs of anger, anxiety and excitement by analyzing a person's voice--not the specific words, but intonations. The proposed approach may be helpful for mental health assessments, market research or even to help call centers improve phone customer relations [10]. Amazon (Bellevue, WA, USA) is also working to improve the ability of its own Alexa AI assistant [11] to interact with users by detecting emotions in their voices. Reference [12] proposed the dataset MOISEI to examine a data fusion model. However, the first research on the multimodal analysis of human emotions was conducted based on two modal parts of information. For example, the usage of a combination of sound and speech modalities is analyzed in [13]. Also, an implementation with a combination of sound and speech in systems with a deep neural network is given in the article in [14]. Based on these works, it was concluded that the fusion of two modal signals provides better accuracy than any unimodal system. There are also many algorithms combining different types of information signals into multimodal data at various stages (early, late) and using other methods of intermodal information extraction. All these implementations are not easy to understand, but they will eliminate the possible loss of information, which is the basis of multimodal emotion data, and give perfect accuracy. At this stage of development, models are studied and improved based on intermodal information in multimodal recognition of the psycho-emotional state. The research aims to develop a model and information system for recognizing human emotions in a stream obtained from a PC or smartphone camera, using different methods of merging channels of image, sound and text, and analysis of the results. To achieve this goal, it is necessary to solve the following tasks:  
* Live generation of data received from a camera and microphone;
* Live generation of data obtained from a recording in the device's memory;
* Live generation of data obtained from the broadcast screen;
* Processing live data to a form that neural networks will accept;
* A mechanism for generating new data to supplement ready-made data sets and create new ones;
* Three neural networks for separate processing of image, sound and text streams;
* Two neural networks for multimodal recognition of emotions;
* Three information systems for recognizing emotions with different ways of merging data streams;
* Interface for ease of use of information systems.