SlowGuess/NAACL_2022_OCR huggingface dataset:
To further demonstrate the effectiveness of our model, we give a case study. We compare the sentiment label predicted based on CLMLF and BERT. As shown in Figure 4, We can find that if we only consider the sentiment of the text, it is difficult to correctly obtain the user's sentimental tendency. For example, for the first data in Figure 4, the meaning of the text is to refer to the image, and
Model	MVSA-Single	MVSA-Multiple	HFM
Acc	F1	Acc	F1	Acc	F1
BERT	0.7111	0.6970	0.6759	0.6624	0.8389	0.8326
ResNet-50	0.6467	0.6155	0.6188	0.6098	0.7277	0.7138
+MLF	0.7111	0.7101	0.7059	0.6849	0.8414	0.8355
+MLF, LBCL	0.7378	0.7291	0.7112	0.6863	0.8489	0.8446
+MLF, DBCL	0.7356	0.7276	0.7153	0.6832	0.8468	0.8422
CLMLF	0.7533	0.7346	0.7200	0.6983	0.8543	0.8487
(a) The text-image Transformer fusion layer
(b) The image Transformer layer
the image expresses a positive meaning. for the second data, if we only observe the text, we find that it may express negative sentiments. If add the image, we find that it is just a joke and actually expresses positive sentiment.
Table 3: Ablation results of our CLMLF
Image	Text	CLMLF	BERT
Why are you feeling despondent? Take the quiz:	Positive	Neutral
Thx for taking me to get cheap slushies ?	Positive	Negative
Car rolls over to avoid real estate sign on Burlington Skyway.	Negative	Neutral
Attention Visualization: We visualize the attention weight of the first head of the TransformerEncoder in the last layer of the Multi-Layer Fusion module. The result of the attention visualization is shown in Figure 5. We can see that for a given keyword, The model can find the target from the image very well and give it more attention weight. This shows that the model aligns the words in the text
with the patch area of the image at a token-level, which plays an important role in the model to fuse text and image features. In particular, for Figure 5b, although "lady" only shows half of the face in the figure, the model still aligns the text and the image very accurately. These indicate that the model aligns the text and image features at token-level according to our assumptions.
Figure 3: Experimental results of different layer of multi-layer fusion module. The solid line indicates the accuracy and the dotted line indicates the F1. The x-axis represents the number of layers of the transformer
(a) The fishing is a little slow but the flowers are vibrant and beautiful.
(b) Kimmy, you're one blessed lady!
Figure 5: Attention visualization of some multimodal sentiment data examples
Figure 4: Example of misclassified by BERT and correctly classified by CLMLF
(c) Martha said for Valentine's Day she wanted a heart shaped pancake for lunch.
(d) It is truly a hilarious, light-hearted read that is a treasure on anyone's bookshelf.
Cluster Visualization: In order to verify that our proposed contrastive learning tasks can help the model to learn common features related to sentiment in multimodal data, we conducted a visualization experiment on the MVSA-Single dataset. The data feature vector of the last layer of the model is visualized by dimensionality reduction. We use the TSNE dimensionality reduction algorithm to obtain a 2-dimensional feature vector and visualize it, as shown in Figure 6, Figure 6a is the visualization of the [CLS] of the Bert-base model, and Figure 6b shows the visualization of the fusion result output from the CLMLF model. From the figure, we can
see that after adding contrastive learning, the distance between positive sentiment and negative sentiment in the vector space is greater, and the degree of data aggregation is more obvious. This shows that the model distinguishes these data in vector space according to common features existing in the same sentimental data. Because the number of neutral sentiment data is relatively small, among the visualization results of the two models, CLMLF's visualization results obviously gather the neutral data together, rather than scattered in the vector space like Bert. All these indicate that adding contrast learning can help the model to learn common features related to sentiment which can improve the performance of the model.
(a) BERT
Figure 6: Cluster visualization of MVSA-Single
(b) Contrastive Learning
5.1 Multimodal Sentiment Analysis
In recent years, deep learning models have achieved promising results for multimodal sentiment analysis. MultiSentiNet (Xu and Mao, 2017) and HSAN (Xu, 2017) use LSTM and CNN to encode texts and images to get hidden representations, then concatenate texts and images hidden representations to fuse multimodal features. CoMN (Xu et al., 2018) uses a co-memory network to iteratively model the interactions between visual contents and textual words for multimodal sentiment analysis. Yu et al. (2019) proposes an aspect sensitive attention and fusion network to effectively model the intra-modality interactions including aspect-text and aspect-image alignments, and the inter-modality interactions. MVAN (Yang et al., 2020) applies interactive learning of text and image features through the attention memory network module, and the multimodal feature fusion module is constructed by using a multi-layer perceptron and a stacking-pooling module. Yang et al. (2021) uses multi-channel graph neural networks with sentiment-awareness which is built based on
the global characteristics of the dataset for multimodal sentiment analysis.
5.2 Contrastive Learning
Self-supervised learning attracts many researchers for its soaring performance on representation learning in the last several years (Liu et al., 2021; Jing and Tian, 2020; Jaiswal et al., 2021). Many models based on contrastive learning have been proposed in both natural language processing and computer vision fields. ConSERT (Yan et al., 2021), SimCSE (Gao et al., 2021), CLEAR(Wu et al., 2020) proposed the application of contrastive learning in the field of natural language processing. MoCo (He et al., 2020), SimCLR (Chen et al., 2020), SimSiam (Chen and He, 2021), CLIP (Radford et al., 2021) proposed the application of contrastive learning in the field of computer vision, and they also have achieved good results in zero-shot learning and few-shot learning. Recently, contrastive learning has been more and more widely used in the field of multimodality. Huang et al. (2021) uses intramodal, inter-modal, and cross-lingual contrastive learning which can significantly improves the performance of video search. Yuan et al. (2021) exploits intrinsic data properties within each modality and semantic information from cross-modal correlation simultaneously, hence improving the quality of learned visual representations.
Compared with the above works, we focus on how to align and fuse the token-level features and learn the common features related to sentiment to further improve the performance of model.
In this paper, we propose a contrastive learning and multi-layer fusion method for multimodal sentiment detection. Compared with previous works, our proposed MLF module performs multimodal feature fusion from the fine-grained token-level, which is more conducive to the fusion of local features of text and image. At the same time, we design learning tasks based on contrastive learning to help the model learn sentiment related features in the multimodal data and improve the ability of the model to extract and fuse features of multimodal data. The experimental results on public datasets demonstrate that our proposed model is competitive with strong baseline models. Especially through visualization, the contrastive learning tasks and multi-layer fusion module we proposed can be ver
ified with intuitive interpretations. In future work, we will incorporate other modalities such as audio into the sentiment detection task.
We are grateful to the anonymous reviewers, meta review and program committee for their insightful comments and suggestions. The work of this paper is funded by the project of National key research and development program of China (No. 2020YFB1406902).
Yitao Cai, Huiyu Cai, and Xiaojun Wan. 2019. Multi-modal sarcasm detection in twitter with hierarchical fusion model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2506-2515.
Peng Chen, Zhongqian Sun, Lidong Bing, and Wei Yang. 2017. Recurrent attention network on memory for aspect sentiment analysis. In Proceedings of the 2017 conference on empirical methods in natural language processing, pages 452-461.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR.
Xinlei Chen and Kaiming He. 2021. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750-15758.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. 2018. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.09501.
Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoc Le. 2020. Randaugment: Practical automated data augmentation with a reduced search space. In Advances in Neural Information Processing Systems, volume 33, pages 18613-18624. Curran Associates, Inc.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at
scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489-500, Brussels, Belgium. Association for Computational Linguistics.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894-6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep residual learning for image recognition. corr abs/1512.03385 (2015).
Lianzhe Huang, Dehong Ma, Sujian Li, Xiaodong Zhang, and Houfeng Wang. 2019. Text level graph neural network for text classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3444-3450, Hong Kong, China. Association for Computational Linguistics.
Po-Yao Huang, Mandela Patrick, Junjie Hu, Graham Neubig, Florian Metze, and Alexander Hauptmann. 2021. Multilingual multimodal pre-training for zero-shot cross-lingual transfer of vision-language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online.
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. 2021. A survey on contrastive self-supervised learning. Technologies, 9(1):2.
Longlong Jing and Yingli Tian. 2020. Self-supervised visual feature learning with deep neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence.
Ramandeep Kaur and Sandeep Kautish. 2019. Multi-modal sentiment analysis: A survey and comparison. International Journal of Service Science, Management, Engineering, and Technology (IJSSMET), 10(2):38-58.
Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. 2021. Transformers in vision: A survey. arXiv preprint arXiv:2101.01169.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746-1751,
Doha, Qatar. Association for Computational Linguistics.
Ayush Kumar and Jithendra Vepa. 2020. Gated mechanism for attention based multi modal sentiment analysis. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4477-4481. IEEE.
Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. 2021. A survey of transformers. arXiv preprint arXiv:2106.04554.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. 2021. Self-supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data Engineering.
Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang, and Heng Ji. 2018. Visual attention model for name tagging in multimodal social media. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1990-1999, Melbourne, Australia. Association for Computational Linguistics.
Teng Niu, Shiai Zhu, Lei Pang, and Abdulmotaleb El Saddik. 2016. Sentiment analysis on multi-view social data. In International Conference on Multimedia Modeling, pages 15-27. Springer.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR.
Rossano Schifanella, Paloma de Juan, Joel Tetreault, and Liangliang Cao. 2016. Detecting sarcasm in multimodal social platforms. In Proceedings of the 24th ACM international conference on Multimedia, pages 1136-1145.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96, Berlin, Germany. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Yequan Wang, Minlie Huang, Xiaoyan Zhu, and Li Zhao. 2016. Attention-based LSTM for aspect-level sentiment classification. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 606-615.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. 2020. Clear: Contrastive learning for sentence representation. arXiv preprint arXiv:2012.15466.
Qizhe Xie, Zihang Dai, Eduard H. Hovy, Thang Luong, and Quoc Le. 2020. Unsupervised data augmentation for consistency training. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Nan Xu. 2017. Analyzing multimodal public sentiment based on hierarchical semantic attentional network. In 2017 IEEE International Conference on Intelligence and Security Informatics (ISI), pages 152-154. IEEE.
Nan Xu and Wenji Mao. 2017. Multisentinent: A deep semantic network for multimodal sentiment analysis. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pages 2399-2402.
Nan Xu, Wenji Mao, and Guandan Chen. 2018. A comemory network for multimodal sentiment analysis. In The 41st international ACM SIGIR conference on research & development in information retrieval, pages 929-932.
Nan Xu, Wenji Mao, and Guandan Chen. 2019. Multi-interactive memory network for aspect based multimodal sentiment analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 371-378.
Nan Xu, Zhixiong Zeng, and Wenji Mao. 2020. Reasoning with multimodal sarcastic tweets via modeling cross-modality contrast and semantic association. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3777-3786.
Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021. ConSERT: A contrastive framework for self-supervised sentence representation transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5065-5075, Online. Association for Computational Linguistics.
Xiaocui Yang, Shi Feng, Daling Wang, and Yifei Zhang. 2020. Image-text multimodal emotion classification via multi-view attentional network. IEEE Transactions on Multimedia.
Xiaocui Yang, Shi Feng, Yifei Zhang, and Daling Wang. 2021. Multimodal sentiment detection based on multi-channel graph neural networks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 328-339.
Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018. *Qanet: Combining local convolution with global self-attention* for reading comprehension. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Jianfei Yu and Jing Jiang. 2019. Adapting bert for target-oriented multimodal sentiment classification. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 5408-5414. International Joint Conferences on Artificial Intelligence Organization.
Jianfei Yu, Jing Jiang, and Rui Xia. 2019. Entity-sensitive attention and fusion network for entity-level multimodal sentiment classification. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:429-439.
Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, and Baldo Faieta. 2021. Multimodal contrastive training for visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6995-7004.
Lin Yue, Weitong Chen, Xue Li, Wanli Zuo, and Minghao Yin. 2019. A survey of sentiment analysis in social media. Knowledge and Information Systems, 60(2):617-663.
Lei Zhang, Shuai Wang, and Bing Liu. 2018a. Deep learning for sentiment analysis: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4):e1253.
Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. 2018b. Adaptive co-attention network for named entity recognition in tweets. In Thirty-Second AAAI Conference on Artificial Intelligence.
Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. Attention-based bidirectional long short-term memory networks for relation classification. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 207-212, Berlin, Germany. Association for Computational Linguistics.
The detailed statistics for the MVSA-Single, MVSA-Multiple and HFM datasets are listed in Table 4. We can see that HFM is a binary classification multimodal sentiment dataset, while MVSA-Single and MVSA-Multiple are three classification multimodal sentiment datasets.
Dataset	Label	Train	Val	Test
MVSA-Single	Positive	2147	268	268
Neutral	376	47	47
Negative	1088	135	135
MVSA-Multiple	Positive	9056	1131	1131
Neutral	3528	440	440
Negative	1040	129	129
HFM	Positive	8642	959	959
Negative	11174	1451	1450
B Aspect Based Multimodal Sentiment
Because CLMLF is designed for sentence-level multimodal sentiment analysis, we have made some minor changes to the input of CLMLF model to adapt to aspect based multimodal sentiment analysis. We change the input form from "[CLS] sentence [SEP]" to "[CLS] sentence [SEP] aspect [SEP]" and no change the input of image modality. Although this change is very simple, CLMLF can work well in aspect based multimodal sentiment analysis tasks and achieves good results.
We use three aspect based multimodal sentiment dataset: Multi-ZOL $^{8}$  (Xu et al., 2019), Twitter-15 (Zhang et al., 2018b) and Twitter-17 $^{9}$  (Lu et al., 2018). The statistics of these datasets are given in Table 5. Compared with the dataset of sentence-level multimodal sentiment analysis, each sentence will have a corresponding aspect attribute. Especially for the Multi-ZOL dataset, each data contains multiple images. And we only randomly select one image for fusion. Although some features are lost, the experimental results show that it is improved compared with the only text modality.
B.2 Results
We compare our model with other baseline models:
Table 4: Number of data for each sentiment category in each dataset
Dataset	Train	Val	Test	Total
Multi-ZOL	22743	2843	2843	28429
Twitter-15	3179	1122	1037	5338
Twitter-17	3562	1176	1234	5972
Table 5: Statistics of the three datasets
- LSTM, a standard sentence-level LSTM model without explicitly considering the aspect. Therefore, this result is also the worst.
- AE-LSTM (Wang et al., 2016), an attention-based LSTM for aspect-level sentiment classification, which uses the attention mechanism to capture the important context information related to the aspect.
- RAM (Chen et al., 2017) is a memory based model, which builds memory on the hidden states of a Bi-LSTM and generates aspect representation based on a Bi-LSTM. Then pays multiple attentions on the memory to pick up important information to predict the final sentiment, by combining the features from different attentions non-linearly.
- MIMN (Xu et al., 2019), the multimodal approach for aspect-level sentiment classification task, which adopts multi-hop memory network to model the interactive attention between the aspect word, the textual context, and the visual context.
- TomBERT (Yu and Jiang, 2019), a multimodal model which borrow the idea from self-attention and design a target attention mechanism to perform target-image matching to derive target sensitive visual representations.
- ESAFN (Yu et al., 2019) proposes an entity-sensitive attention and fusion network which captures the intra-modality dynamics by leverages an effective attention mechanism to generate entity-sensitive textual and visual representations. And uses visual attention mechanism to learn the entity-sensitive visual representation. Moreover, ESAFN further fuses the textual and visual representations with a bilinear interaction layer.
Table 6 illustrates the performance comparison of our CLMLF mdoel with the baseline methods. We use Macro-F1 and ACC as the evaluation metrics for all datasets. The experimental results show
$^{8}$ https://github.com/xunan0812/MIMN
$^{9}$ https://github.com/jefferyYu/TomBERT
Modality	Model	Multi-ZOL	Twitter-15	Twitter-17
Acc	F1	Acc	F1	Acc	F1
Text	LSTM	0.5892	0.5729	0.6798	0.5730	0.5592	0.5169
AE-LSTM	0.5958	0.5895	0.7030	0.6343	0.6167	0.5797
RAM	0.6018	0.5968	0.7068	0.6305	0.6442	0.6101
BERT	0.6959	0.6868	0.7387	0.7023	0.6848	0.6553
Multimodal	MIMN	0.6159	0.6051	0.7184	0.6569	0.6588	0.6299
ESAFN	-	-	0.7338	0.6737	0.6783	0.6422
TomBERT	-	-	0.7715	0.7175	0.7034	0.6803
CLMLF	0.7452	0.7075	0.7811	0.7460	0.7098	0.6913
Table 6: Experimental results of different models on aspect based datasets
Model	Multi-ZOL	Twitter-15	Twitter-17
Acc	F1	Acc	F1	Acc	F1
BERT	0.6959	0.6868	0.7387	0.7023	0.6848	0.6553
+MLF	0.7301	0.6897	0.7424	0.7017	0.6848	0.6579
+MLF, LBCL	0.7336	0.6953	0.7715	0.7311	0.6969	0.6790
+MLF, DBCL	0.7347	0.7015	0.7445	0.6964	0.6921	0.6722
CLMLF	0.7452	0.7075	0.7811	0.7460	0.7098	0.6913