# Deep Multimodal Emotion Recognition on Human Speech: A Review
## 5 Conclusions and Future Challenges
\begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline \hline
**Model** & **Reference** & **Validated by** & **Modalities** & **Validation** & **Happy** & **Sad** & **Angry** & **Neutral** \\ \hline  & & & & & **A** & **F1** & **A** & **F1** & **A** & **F1** & **A** & **F1** \\ \hline CTC + RAVEN & [69] & [70] & \{A, T, V\} & \begin{tabular}{c} Train-Val-Test \\ 3-1-1 Sessions \\ \end{tabular} & 77.0 & 76.8 & 67.6 & 65.6 & 65.0 & 64.1 & 62.0 & 59.5 \\ CTC + MCTN & [58] & [70] & \{A, T, V\} & \begin{tabular}{c} Train-Val-Test \\ 3-1-1 Sessions \\ \end{tabular} & 80.5 & 77.5 & 72.0 & 71.7 & 64.9 & 65.6 & 49.4 & 49.3 \\ MuIT & [70] & [70] & \{A, T, V\} & \
\begin{tabular}{c} Train-Val-Test \\ 3-1-1 Sessions \\ \end{tabular} & 84.8 & 81.9 & 77.7 & 74.1 & 73.9 & 70.2 & 62.5 & 59.7 \\ MTAG & [71] & [71] & \{A, T, V\} &
\begin{tabular}{c} Train-Val-Test \\ 3-1-1 Sessions \\ \end{tabular} & - & **86.0** & - & **79.9** & - & **76.7** & - & **64.1** \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Multimodal emotion recognition evaluation results for unaligned sequential input data. The works in which a specific multimodal architecture was validated are indicated under the “Validated by” column, while the type of validation is marked in the “Validation” column. For architectures that were not created to work in an unaligned manner, the CTC method was used. Apart from the aforementioned challenges that will result in more robust and better statistically evaluated systems, more efficient methods need to be adopted for signal representation. More specifically, in the task of multimodal speech emotion recognition, a few specific low-level modality representation techniques have been established. In particular, for the aural modality, signal-analysis-based features that are associated with both the spectral and temporal domains are used. As for the textual and visual modalities, basic word embeddings are used for the former, while traditional facial features are used for the latter. However, the development of deep learning techniques has resulted in better representation capabilities for all three of these modalities. Thus, multimodal architectures need to be in line with the state-of-the-art techniques for unimodal representations and find efficient ways to combine them by taking into account the temporal dimension. Finally, each modality can be enriched with new representations that also take into account parts of the information that have not been so widely used until now. For example, despite the fact that body language is strongly correlated with expressed emotions, the only information that is captured from the visual modality is that of facial expressions. Last but not least, one more concept that is absent in the literature of multimodal emotion recognition is unsupervised representation. In other machine learning application domains, there are many powerful unsupervised representation methods, such as BERT [41] for text, wav2 vec 2.0 [31] for audio and MoCo [92] for vision, that are adopted in an application-independent manner. Several well-defined datasets have been created for the task of emotion recognition, and research that combines them in order to generate generic multimodal emotion-oriented representations may result in a exponential growth of the use of emotional analysis in other application domains. As an example, an \"emotional\" multimodal embedding methodology could be used in a speaker verification system to improve the robustness of the system to the emotional state of the speaker. Conceptualization, P.K. and T.G. ; methodology, P.K. ; validation, T.G. ; investigation, P.K. and T.G. ; visualization, P.K. ; supervision, T.G. ; writing--original draft preparation, P.K. and T.G. ; writing--review and editing, P.K. and T.G. Both authors have read and agreed to the published version of the manuscript. This research received no external funding. The authors declare no conflict of interest.