SlowGuess/ACL_2020_OCR huggingface dataset:
Evaluation Metrics. The same as Liu et al. (2018); Zadeh et al. (2018b), we record our experimental results in two forms: multi-class classification and regression. For multi-class classification, we report Weighted F1 score and multi-class accuracy Acc-  $k$ , where  $k \in \{2,3,5\}$ . For regression, we report Mean Absolute Error (MAE) and Pearson correlation (Corr). Except for MAE, higher values denote better performance for all metrics.
In this section, we present and discuss the experimental results of the research questions introduced in Section 5.
5.3.1 Comparison with Baselines.
We compare three new methods with the aforementioned baselines. In this part, we only consider the
multimodal evaluation results though new methods are multi-task. Results are shown in Table 2. Compared with single-task models, multi-task models have better performance in most of evaluation metrics. In particular, all three improved models (MLF-DNN, MLFM, and MTFN) have promotion significantly compared to corresponding original models (LF-DNN, LFM, and TFN) in all evaluation metrics except for Acc-5. The above results demonstrate that the introduction of independent unimodal annotations in multimodal sentiment analysis can significantly improve the performance of existing methods. Also, we find that some methods, such as MULT, that perform well on existing public datasets while they are not satisfactory on SIMS. It further illustrates that designing a robust, cross-lingual multimodal sentiment analysis model is still a challenging task, which is also one of our motivations for proposing this dataset.
5.3.2 Unimodal Sentiment Analysis.
Due to the independent unimodal annotations in SIMS, we conducted two sets of experiments for unimodal sentiment analysis. In the first set of experiments, we use real unimodal labels to verify the model's ability of performing unimodal sentiment analysis. In the second set of experiments, we use multimodal labels instead of unimodal labels to verify the ability of predicting the true emotions of speakers when there is only unimodal information.
Results are shown in Table 4. Firstly, in the same unimodal task, the results under unimodal labels are better than those under multimodal labels. But the former cannot reflect the actual sentimental state of speakers. Secondly, under multimodal annotations, the performance with unimodal information only is lower than using multimodal information in Table 2. Hence, it is inadequate to perform sentiment analysis using unimodal information only due to the inherent limitations of unimodal information.
5.3.3 Representations Differences.
Another motivation for us to propose CH-SIMS is that we think the unimodal representation differences will be greater with independent unimodal annotations. We use t-SNE (Maaten and Hinton, 2008) to visualize intra-modal representations learned in original models (LF-DNN, TFN, and LMF) and new models (MLF-DNN, MTFN, and MLMF), shown as Figure 4. It is relatively obvious that new models learn more distinctive unimodal
LF-DNN
TFN
MLF-DNN
MTFN
MLMF
Figure 4: Visualization in Unimodal Representations. In each subfigure, red, green, and blue points represent the unimodal representations in text, audio, and video, respectively. The first row shows the learned representations from models with the multimodal task only. The second row shows the learned representations from multi-task models. The two subgraphs in the same column contrast each other
representations compare to original models. Therefore, unimodal annotations can help the model to obtain more differentiated information and improve the complementarity between modalities.
In this section, we compare the difference in the effects of combining different unimodal tasks on multimodal sentiment analysis. We aim to further explore the influence on multimodal sentiment analysis with different unimodal tasks. Furthermore, we reveal the relationship between multi-task learning and multimodal sentiment analysis.
We conducted multiple combination experiments to analyze the effects of different unimodal subtasks on the main multimodal task. In this part, we only report the results in MLF-DNN. Results are shown in Table 5. The results show that in the case of partial absence of three unimodal subtasks, the performance of the multimodal task has not significantly improved, or even damaged. Two factors may cause an adverse effect in multimodal learning, including the consistency between different unimodal representations and the asynchrony of learning in different tasks. The former means that unified annotations guide the representations to be similar and lack complementarity in different modalities. The latter means that the learning process in different tasks is inconsistent. Taken tasks
Tasks	Acc-2	F1	MAE	Corr
M	80.04	80.40	43.95	61.78
M, T	80.04	80.25	43.11	63.34
M, A	76.85	77.28	46.98	55.16
M, V	79.96	80.38	43.16	61.87
M, T, A	80.88	81.10	42.54	64.16
M, T, V	80.04	80.87	42.42	60.66
M, A, V	79.87	80.32	43.06	62.95
M, T, A, V	82.28	82.52	40.64	64.74
Table 5: (%) Results for multimodal sentiment analysis with different tasks using MLF-DNN. "M" is the main task and "T, A, V" are auxiliary tasks. Only the results of task "M" are reported.
"M, A" as an example, the sub-network of subtask "A" is supervised by multimodal loss and unimodal loss. In contrast, subtask "T" and subtask "V" are supervised by their unimodal loss only. It means the "A" is learned twice while the "T" and the "V" are learned once only during an training epoch. Therefore, the introduction of unimodal tasks will reduce the consistency of the representation and strengthen the complementarity, but will also cause the asynchrony. As more unimodal tasks are introduced, the positive effects of the former gradually increase, and the negative effects of the latter gradually decrease. Finally, when all unimodal tasks are added, the negative effect of the latter is almost dis
appearing. Finally, the performance of the model with tasks "M, T, A, V" reaches a peak.
In this paper, we propose a novel Chinese multimodal sentiment analysis dataset with independent unimodal annotations and a multimodal multi-task learning framework based on late-fusion methods. We hope that the introduction of CH-SIMS will provide a new perspective for researches on multimodal analysis. Furthermore, we conduct extensive experiments on discussing unimodal, multimodal, and multi-task learning. Lastly, we summarize our overall findings as follows:
- Multimodal labels cannot reflect unimodal sentimental states always. The unified multimodal annotations may mislead the model to learn inherent characteristics of unimodal representations.
- With the help of unimodal annotations, models can learn more differentiated information and improve the complementarity between modalities.
- When performing multi-task learning, the asynchrony of learning in different subtasks may cause an adverse effect on multimodal sentiment analysis.
In the future, we will further explore the connection between multimodal analysis and multi-task learning and incorporate more fusion strategy, including early- and middle-fusion.
This paper is founded by National Natural Science Foundation of China (Grant No: 61673235) and National Key R&D Program Projects of China (Grant No: 2018YFC1707605). We would like to thank the anonymous reviewers for their valuable suggestions.
Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2018. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2):423-443.
Tadas Baltrusaitis, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency. 2018. Openface 2.0: Facial behavior analysis toolkit. In 2018 13th IEEE
International Conference on Automatic Face & Gesture Recognition (FG 2018), pages 59-66. IEEE.
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335.
Erik Cambria, Devamanyu Hazarika, Soujanya Poria, Amir Hussain, and RBV Subramanyam. 2017. Benchmarking multimodal sentiment analysis. In International Conference on Computational Linguistics and Intelligent Text Processing, pages 166-179. Springer.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369-376. ACM.
Wenzhong Guo, Jianwen Wang, and Shiping Wang. 2019. Deep multimodal representation learning: A survey. IEEE Access, 7:63373-63394.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Runnan Li, Zhiyong Wu, Jia Jia, Jingbei Li, Wei Chen, and Helen Meng. 2018. Inferring user emotive state changes in realistic human-computer conversational dialogs. In 2018 ACM Multimedia Conference on Multimedia Conference, pages 136-144. ACM.
Ya Li, Jianhua Tao, Linlin Chao, Wei Bao, and Yazhu Liu. 2017. Cheavd: a chinese natural emotional audio-visual database. Journal of Ambient Intelligence and Humanized Computing, 8(6):913-924.
Bing Liu and Lei Zhang. 2012. A survey of opinion mining and sentiment analysis. In *Mining text data*, pages 415-463. Springer.
Wu Liu, Tao Mei, Yongdong Zhang, Cherry Che, and Jiebo Luo. 2015. Multi-task deep visual-semantic embedding for video thumbnail selection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3707-3715.
Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, Amir Zadeh, and LouisPhilippe Morency. 2018. Efficient low-rank multimodal fusion with modality-specific factors. arXiv preprint arXiv:1806.00064.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579-2605.
Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt McVicar, Eric Battenberg, and Oriol Ni eto. 2015. librosa: Audio and music signal analysis in python. In Proceedings of the 14th python in science conference, volume 8.
Louis-Philippe Morency, Rada Mihalcea, and Payal Doshi. 2011. Towards multimodal sentiment analysis: Harvesting opinions from the web. In Proceedings of the 13th international conference on multimodal interfaces, pages 169-176. ACM.
Bo Pang, Lillian Lee, et al. 2008. Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(1-2):1-135.
Verónica Pérez-Rosas, Rada Mihalcea, and Louis-Philippe Morency. 2013. Utterance-level multimodal sentiment analysis. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 973-982.
Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Mazumder, Amir Zadeh, and Louis Philippe Morency. 2017. Multi-level multiple attentions for contextual multimodal sentiment analysis. In 2017 IEEE International Conference on Data Mining (ICDM), pages 1033-1038. IEEE.
Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2018. Meld: A multimodal multi-party dataset for emotion recognition in conversations. arXiv preprint arXiv:1810.02508.
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6558-6569, Florence, Italy. Association for Computational Linguistics.
Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2018. Learning factorized multimodal representations.
Jennifer Williams, Steven Kleinegesse, Ramona Comanescu, and Oana Radu. 2018. Recognizing emotions in video using multimodal DNN feature fusion. In Proceedings of Grand Challenge and Workshop on Human Multimodal Language (Challenge-HML), pages 11-19, Melbourne, Australia. Association for Computational Linguistics.
Martin Wöllmer, Felix Weninger, Tobias Knaup, Björn Schuller, Congkai Sun, Kenji Sagae, and Louis-Philippe Morency. 2013. Youtube movie reviews: Sentiment analysis in an audio-visual context. IEEE Intelligent Systems, 28(3):46-53.
Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. arXiv preprint arXiv:1707.07250.
Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018a. Memory fusion network for multiview sequential learning. In Thirty-Second AAAI Conference on Artificial Intelligence.
Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:1606.06259.
AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018b. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2236-2246.
Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. 2016a. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters, 23(10):1499-1503.
Wenlu Zhang, Rongjian Li, Tao Zeng, Qian Sun, Sudhir Kumar, Jieping Ye, and Shuiwang Ji. 2016b. Deep model based transfer and multi-task learning for biological image analysis. IEEE transactions on Big Data.
Yu Zhang and Qiang Yang. 2017. A survey on multitask learning. arXiv preprint arXiv:1707.08114.