# Model for Determining the Psycho-Emotional State of a Person Based on Multimodal Data Analysis
## 4 Results
Figure 9: TSNE for FER (**a**), SER (**b**), TER (**c**)—training dataset. Based on these three conclusions, we are able to qualitatively analyze the results of testing in \"live\" mode. Figure 11: TSNE for MMERs (**a**), MMERm (**b**)—testing dataset. Figure 12: TSNE for FER (**a**), SER (**b**), TER (**c**)—testing dataset. First, Table 1 shows the resulting accuracy of five neural networks. The highest validation accuracy based on multimodal data is obtained for MMERS. However, multimodal accuracy is still worse than the accuracy for unimodal FER. FER, SER and TER were trained on a unimodal separated dataset, then the pretrained neural networks are used in merged neural networks. Table 2 shows the metrics of testing on validation data. Metric values will be presented as \"weighted avg\". However, metrics \"micro avg\", \"macro avg\" and \"samples avg\" are also available for analysis. The best result is obtained for FER (video modality) and the worst result is obtained for SER (audio modality). The difference in accuracy across modalities is explained by the difference in the quality of the datasets. So, for images, a sequence of three images is analyzed. A convolutional neural network works well for recognizing individual images. Moreover, accuracy increases with combined results of CNN and LSTM for sequential images. Speech recognition is performed for a dataset consisting of audio files 4 s long, due to which the end of the sentence to determine the emotion is not always correct. The same situation exists for the text dataset. At the same time, the combination of three modalities makes it possible to significantly improve the result. For a better understanding, we display this data in the form of a bar chart in Figure 13. \begin{table}
\begin{tabular}{l l l l l l} \hline \hline  & \multicolumn{2}{l}{**MMERe**} & & & & \\ \cline{2-5}  & **FER** & **SER** & **TER** & & \\ \hline val\_accuracy & 0.9245 & 0.4363 & 0.6416 & 0.7000 & 0.7347 \\ \hline val\_loss & 0.4402 & 0.9542 & 0.7048 & 0.4647 & 0.6396 \\ \hline train\_accuracy & 1.0000 & 0.6544 & 0.8209 & 0.9583 & 0.9774 \\ \hline train\_loss & 0.0267 & 0.8468 & 0.4675 & 0.0732 & 0.05832 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Table of neural network training results. \begin{table}
\begin{tabular}{l l l l l} \hline \hline  & \multicolumn{2}{l}{**MMERe**} & & & \\ \cline{2-5}  & **FER** & **SER** & **TER** & \\ \hline precision & 0.92 & 0.56 & 0.64 & 0.92 & 0.90 \\ \hline recall & 0.92 & 0.56 & 0.64 & 0.90 & 0.89 \\ \hline f1-score & 0.92 & 0.56 & 0.64 & 0.91 & 0.89 \\ \hline support & 106 & 319 & 16377 & 120 & 720 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Table of test results of networks after training. Figure 13: Bar chart of average training results. The value of such metrics can be obtained for each emotion separately. For a better understanding, we display this data in the form of a bar chart in Figure 14.  
In addition to validation training, it is important to analyze live test results, as these are important in our system of multimodal recognition of a person's psycho-emotional state in the data stream from the camera and microphone or recording or broadcasting the user's screen. First, we present the results of various metrics with averaging \"weighted avg\" (Table 3). The results of the same metrics for each emotion separately are given in Figure 15.  
Happy emotion is the best recognized by FER, SER, MMERm and MMERs. Neutral emotion is the hardest emotion for recognition by SER and TER. We will also show an alternative version of the visualization of test results for each emotion in Figure 16.  
Next, the training time for different numbers of emotions and accuracy were analyzed. All neural networks were trained on the same dataset. Figure 17 presents time dependence of number of emotions. The maximum number of emotions for multimodal data is six because the same number of emotions is used in the FER dataset.