# Model for Determining the Psycho-Emotional State of a Person Based on Multimodal Data Analysis
## 3 Materials and Methods  
### Data Preparation  
The solution implemented in the paper is based on the combination of simple unimodal methods for emotion recognition into one multimodal method. This solution allows a reasonably accurate result to be obtained, for example, in recognizing a person's psycho-emotional state while using the banking system [25]. Also, developing such an information system will make it possible to understand the advantages of multimodal recognition of emotions. The most straightforward 3 options of the model and information system that can be implemented (Figure 1) are the following:  
* Merger of three information streams at the stage when the probability is obtained for three models separately. In this case, we take a constant weighted sum of all three results (Figure 1a). * Stream merging at the stage when each model has formed a certain set of characteristics for its data stream. In this case, we combine all these characteristics into one array and submit it for further processing with a typical output (Figure 1b). * Data merging at the beginning of the neural network immediately after pre-processing (Figure 1c). Multi-modal emotion recognition (MMER) system merges text, sound and images. The data flow in the system is realized not only from the camera and microphone, but also from the saved recording in the device memory and the screen broadcast. Based on the analyzed requirements, we will use CNN [33] and LSTM [34] models of neural networks for three types of data and different ways of merging them. We can identify three points where new data arrives. The first moment is when we read data from the camera and microphone (or from a recording or broadcast of the screen) and submit it for processing. The second point is when the data is fed to the input of the neural network for the learning process. The third point is when pre-processed data is fed to a trained network to predict human emotions. Based on characteristic of supervised learning in [35], we have to enter two data streams, x and y. In our case, y is the probability vector that the emotion belongs to a certain category. And x is either data of one type or an array of data of different types. This is also called the classification problem. Consider the data for a text emotion recognition (TER) model, i.e., to recognize emotion in text. The data in these datasets are taken from the Kaggle resource [36]. There are about 5-15 words in one fragment of text. The total number of such fragments in the initial data set is 47,291. Figure 1: Diagrams of systems with different methods of merging (**a**) merging in the end; (**b**) merging in the middle; (**c**) merging in the start. Next, consider a similar folder for speech emotion recognition (SER). This is a RAVDESS dataset taken from the Kaggle resource [37]. Files are organized in folders based on different actors who voice emotions. The average recording time in these files is 3-6 min. The original data set consists of 1440 files. Now let us look at a similar folder for training the neural network facial emotion recognition (FER). This is a data set CK + 48 with 5 emotions [38] taken from the resource Kaggle. Faces are manually cropped in that dataset [39]. Each folder has a sequence of 3 images, which is required for proper training of our LSTM model. Images contain the faces of different people. The size of the image is 48*48 in grayscale. The total number of images in this initial data set is 765. The Fer13 dataset [40] contains 8 emotions. This dataset can be used for testing the efficiency of the proposed model. The amount of data in the sets can be easily increased due to the implemented functionality of data regeneration from other sets or from the live data stream from the camera and microphone (or from recording or broadcasting the screen). The preprocessing is used for that. At the output of the camera, we have an image in three RGB colors.