# Model for Determining the Psycho-Emotional State of a Person Based on Multimodal Data Analysis
## 4 Results  
The system was developed based on the following technologies algorithms and frameworks:  
* CNN, LSTM
* Dropout, MaxPooling, Flatten, Dense, Reshape, GlobalMaxPool2D, BatchNormalization, Bidirectional, TimeDistributed, Embedding;
* Nadam;
* Relu, softmax, elu;
* EarlyStopping, ReduceLROnPlateau;
* Accuracy, categorical_crossentropy;
* Precision, recall, f1-score, support. In addition:  
* PCA, TSNE;
* OpenCV 4.8.1 (convertScaleAbs, LUT);
* Ffmpeg 0.2.0 -python (ffmpeg.input);
* Librosa 0.10.1 (feature.mfcc);
* SpeechRecognition (speech_recognition.recognize_google);
* Tokenizer (texts to sequences). For results evaluation, the following metrics are used: accuracy, categorical_crossentropy, precision, recall, f1-score. Accuracy is the percentage of correctly classified data (_tp_--true positive, _tn_--true negative, _fp_--false positive, _fn_--false negative):  
\[\text{A}=\frac{tp+tn}{tp+tn+fp+fn}.\]  
Figure 8: The structure of the MMERs neural network. Categorical cross-entropy is a loss function that calculates the difference between two probability distributions: the observed distribution and the predicted probability distribution. In the case of classification, the input is fed to the input of the model, which generates a vector of distribution probabilities across all classes. The next step is to compare this vector with the vector of actual values and calculate the cross-entropy between the two vectors. The basic idea behind categorical cross-entropy is that we penalize a model if its predicted probabilities differ from the actual values. Therefore, the goal of optimization is to reduce this loss by adjusting the model parameters. Because neural networks are used not only in the learning process, where logits immediately hit the evaluation function but also to show the percentage of predicted emotion, the result normalized by the Softmax layer is more presentable and understandable. If we consider the context of the study of structures on metrics, the softmax layer can be omitted and specify the CrossEntropyloss function with the from_logits = true parameter, in which case the internal code would do the same as the softmax layer. Therefore, this decision has no effect on the result. The separate layer was made for convenience in our context. Precision is the proportion of relevant instances among the extracted instances:  
\[P=\frac{tp}{tp+fp}.\]  
Recall is the proportion of relevant instances that were retrieved:  
\[R=\frac{tp}{tp+fn}.\]  
\(F1\)-score combines precision and recall and takes their harmonic mean. It is used to compare the effectiveness of classifications:  
\[F1=\frac{2\times(P\times R)}{P+R}.\]  
Python 3 Google Compute Engine backend (GPU) was used for experiments performing. To begin, let us look at detailed results for training and testing using six emotional states. For qualitative assessment, TSNE derivation is also implemented for the characteristics formed in each network at the penultimate layer of the neural network during prediction. The program uses \"PCA\" and \"TSNE\" implementations for this. For example, on the validation data for five neural networks, we will have the graphs shown in Figures 9 and 10. Therefore, the cluster density for FER is the best compared with the rest of the unimodal models. The SER model demonstrates clusters overlapping. MMERm and MMERs demonstrate appropriate cluster density too. And finally, we visualize the TSNE graphs for three architectures for test dataset (Figures 11 and 12). TSNE converts similarities between data points to joint probabilities and tries to minimize the Kullback-Leibler divergence between the joint probabilities of the low-dimensional embedding and the high-dimensional data (based on scikit-learn version 0.16.1 documentation [43]). The obtained result is the same. The best result is for MMERm, MMERs and FER. SER presents clusters overlapping. Anger emotion is well recognized based on MMERm. Figure 10: TSNE for MMERm (**a**) and MMERs (**b**)—training dataset.