# Model for Determining the Psycho-Emotional State of a Person Based on Multimodal Data Analysis
## 4 Results
\begin{table}
\begin{tabular}{l l l l l l} \hline \hline  & \multicolumn{2}{l}{**MMERe**} & & & & \\ \cline{2-6}  & **FER** & **SER** & **TER** & **MMERm** & **MMERs** \\ \hline precision & 0.78 & 0.54 & 0.67 & 0.71 & 0.62 \\ \hline recall & 0.66 & 0.48 & 0.63 & 0.62 & 0.58 \\ \hline f1-score & 0.715 & 0.51 & 0.65 & 0.66 & 0.60 \\ \hline support & 67 & 31 & 29 & 62 & 52 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Table of test results of networks after testing. Figure 14: Bar chart of training results for each emotion. Figure 16: Drawing test results for each emotion. Figure 15: Bar chart of test results for each emotion. Figure 17: Training time dependence from number of emotions. More detailed information is presented in Tables 4 and 5. \begin{table}
\begin{tabular}{c c c c c c c c c c c c c} \hline \hline  & & & & & & & & & & & & & & & \\ \hline FER & - & - & - & - & - & - & - & - & - & - & 310.12 & 1.0/0.92 & 464 \\ SER & - & - & - & - & 36 & 0.67/0.47 & 138 & 34.74 & 0.71/0.55 & 141 & 52.03 & 0.72/0.55 & 250 \\ TER & 725.43 & 0.75/0.66 & 48 & 634 & 0.75/0.65 & 45 & 567.93 & 0.74/0.66 & 45 & 547.73 & 0.74/0.65 & 45 \\ MMER-S & - & - & - & - & - & - & - & - & - & 29.51 & 1.00/0.97 & 78 \\ MMER-M & - & - & - & - & - & - & - & - & - & 195.6 & 0.99/0.85 & 18 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Accuracy and training time for different numbers of emotions (from nine to six emotions). \begin{table}
\begin{tabular}{c c c c c c c c c c c c c} \hline \hline  & & & & & & & & & & & & & \\ \hline FER & 96.8 & 0.87/0.86 & 157 & 125.61 & 0.99/0.88 & 222 & 140.25 & 1.0/0.85 & 347 & 69.57 & 1.0/1.0 & 196 \\ SER & 41.71 & 0.73/0.64 & 235 & 55.85 & 0.77/0.66 & 346 & 22.45 & 0.80/0.75 & 176 & 2.84 & 0.59/0.75 & 21 \\ TER & 421.23 & 0.74/0.65 & 36 & 369.32 & 0.86/0.78 & 38 & 249.9 & 0.97/0.94 & 40 & 84.88 & 0.98/0.95 & 66 \\ MMERs & 15.28 & 0.96/0.90 & 43 & 18.94 & 1.00/0.94 & 71 & 13.32 & 1.00/0.89 & 57 & 7.062 & 1.00/0.94 & 39 \\ MMERm & 211.58 & 0.98/0.82 & 22 & 170.03 & 0.99/0.95 & 24 & 105.82 & 1.00/0.95 & 16 & 102.76 & 0.99/1.00 & 26 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Accuracy and training time for different numbers of emotions (five to two emotions). The list of emotions is presented below:  
* \"sad\", \"anger\", \"surprise\"  
From Tables 4 and 5, we can make a conclusion about the dependence of learning time on the number of emotions. Also, the accuracy of recognition slightly decreases with the increase in the number of emotions. Although for six emotions, the accuracy is higher than for five. This situation is explained by the features of the data sets and the number of emotions that were basically presented in them. The difference in train and validation accuracy for different numbers of emotions can be explained in the following way. Some emotions are similar and that is why it is very hard to recognize them based on only one modality. From the other side, MMERs looks the best in comparison with the rest of the models. The next experiment was conducted based on size of images. The purpose of the experiment was to test the hypothesis of whether a larger image size would affect the accuracy of emotion classification. To conduct the experiment, two options for changing the data were used. The first option was to artificially increase the image size of the original CK + 48 dataset to 192*192. The second way was to use new data obtained from a real data stream (noted with symbol +) in which the frame size was 192*192. Also, for comparison, the new images were reduced to 48*48. Accordingly, for the analysis of new data, the FER architecture in Figure 4 has also been resized to fit the image. The change in the training time of the FER neural network depending on the image size was also analyzed. Early stopping is used to increase the efficiency of training neural networks and to avoid overfitting. On other hand, one of the drawbacks of early stopping is that it can be sensitive to the choice of the validation set, the criterion, and the threshold or the patience parameter.