SlowGuess/ACL_2020_OCR huggingface dataset:
We use F1-score and Accuracy as the evaluation metrics. Table 3 shows the comparative results of our model and these unimodal baseline models. Though ResNet demonstrates the superior performance in many image classification tasks, it performs relatively poor in sarcasm detection task. It is because that the sarcasm intention or visual contrast context in the image is usually unobvious. CNN and BiLSTM just treat the sarcasm detection task as a text classification task, ignoring the contextual contrast information. Thus, their performances are worse than MIARN, which focuses on textual context to model the contrast information between individual words and phrases. However, due to the nature of short text, relying on textual information is often insufficient, especially in multimodal tweets where cross-modality context relies the most important role. Our D&R Net performs better than unimodal baselines, demonstrating the usefulness of modeling multiple modality information in providing additional cues through reasoning contextual contrast and association.
To evaluate the performance of each component used in our D&R Net, we conduct the detailed ablation studies on various variants of our model. The ablation results are shown in Table 4.
In general, we find those variants underperform our model. The most obvious declines come from the direct removal of our two core modules, D-Net and R-Net (see row 1, 3). Comparing these two variants, we find that removing D-Net has greater performance drop than removing R-Net. This suggests that modeling the cross-modality contrast in D-Net is more useful than cross-modality association in R-Net. After removing the D-Net, the model only accepts the text and ANPs inputs. Thus we
Table 3: Comparative results with unimodal baselines
Variant	Evaluation Metric
F1	Acc	△F1	△Acc
D&R Net	80.60	84.02	-	-
1 - D-Net	77.63	82.27	-2.97	-1.75
2 + ⊕Image	79.10	82.73	-1.50	-1.29
3 - R-Net	79.90	83.10	-0.70	-0.92
4 + ⊕ANPs	78.68	83.11	-1.92	-0.91
5 - ANP, +Attribute	79.52	83.12	-1.08	-0.90
6 - ANP-P.F., +MaxPool	79.80	83.27	-0.80	-0.75
7 - ANP-P.F., +AvgPool	79.86	83.42	-0.74	-0.60
Table 4: Ablation results of our D&R Net
further incorporate image information via directly concatenating image encoding in the final fusion layer (see row 2). The improvement compared with - D-Net shows the effectiveness of using image modality for multimodal sarcasm detection. Similarly, we also add the representation of ANPs to the fusion layer after removing the R-Net module (see row 4). However, the performance unexpectedly continues to decrease. One possible reason for this is that the fusion of ANPs affects the original decomposition results in spite of using triple inputs. It is worth mentioning that replacing our ANPs with noun attributes used in (Cai et al., 2019) underperforms our model (see row 5). This result indicates that ANPs are more useful in modeling semantic association between image and text compared with noun attributes. It is because that the adjectivenoun words in ANPs are more semantically informative than noun-only words. Finally, we notice that our ANP-probability fusion (i.e. ANP-P.F.) strategy provides a means for obtaining reasonable performance compared with standard pooling operations, MaxPool and AvgPool (see row 6, 7), with ANP-probability weighted average performing the best.
In this section, we provide case studies through several practical examples to illustrate that our D&R Net really learns to reason multimodal sarcastic tweets with interpretability.
4.6.1 Illustrative Examples
Fig.3 shows some multimodal non-sarcasm and sarcasm examples that our model correctly predicts. For those text-only or image-only models, it's almost impossible to detect the sarcasm intention of Fig.3a and 3b. We also show the results of the extracted ANPs from each image and these ANPs actually provide useful information for sarcasm
Happy Spring! Loving all the blossoming flowers happening here! So beautiful!
Weather's looking amazing today. Walmart parking lot at sunrise.
ANPs
1.empty space, 0.97126
2.heavy snow, 0.96964
3.cloudy mountains, 0.96932
4.misty winter, 0.96698
5.icy fog, 0.95835
(a) Sarcasm
1.empty space, 0.94933
2.lonely car, 0.9417
3.lonely road, 0.93728
4.wet road, 0.92808
5.empty street, 0.92797
1.magnificent sunrise, 0.9947
2.amazing sunset, 0.99248
3.clear night, 0.99218
4.incredible sunset, 0.99183
5.great sunset, 0.98959
(c) Non-Sarcasm
1.cute girls, 0.96865
2.evil queen, 0.96622
3.energetic performance, 0.96161
4.traditional dress, 0.95733
5.favorite band, 0.95693
(d) Non-Sarcasm
Figure 3: Examples of multimodal non-sarcasm and sarcasm tweets with extracted ANPs results.
(a) What a beautiful eclipse! #EclipseDay
(b) Can't wait to spend several hours every day in the next few weeks with these lovely books. #lawstudentproblems
Figure 4: ANP-aware cross-modality attention visualization of multimodal sarcasm examples.
detection. For example, the ANPs heavy snow, cloudy mountains, minsty winter of Fig.3a show the great conflict with text word 'Spring', conveying the strong intention of sarcasm. In addition, our extracted ANPs are more semantically meaningful than the noun-only attributes used in (Cai et al., 2019). The wet road and empty street are more informative than noun-only words road and street in Fig.3b. The cute girls and energetic performance are more in line with the text words 'so beautiful' compared with noun-only words girls and performance in Fig.3d to discriminate between sarcasm and non-sarcasm.
4.6.2 Attention Visualization
Our proposed ANP-aware cross-modality attention mechanism explicitly calculates the cross interactive attention between text words and image ANPs, providing the explainable reasoning evidence for sarcasm detection. We further illustrate this attention mechanism by visualizing its outputs on two multimodal sarcastic tweets in Fig.4. The results show that our proposed attention mechanism works well for multimodal sarcasm detection by explicitly identify the relationship between image regions and text words. For instance, in Fig.4a, the user satirically mentions eclipse for too many clouds covering the sun. Our D&R Net accurately detects sarcasm intention via focusing on the text words
'ecipse', '!', 'EclipseDay' with multiple visual semantic ANP views: stormy, fluffy, lovely and rainy clouds. In Fig.4b, our model pays more attention to the textual phrase 'these lovely books' with stupid sign, strange sign, and bad sign ANPs which refer to the emoji in the attached image. Consequently, it is easy for our model to detect the sarcasm intention that the books are NOT 'lovely' at all.
In this paper, we identify the essential research issue in multimodal sarcasm detection. To model the cross-modality contrast in the associated context of multimodal sarcastic tweets, we propose the D&R Net to represent the commonality and discrepancy between image and text and multi-view semantic associations in cross-modality context. Our model is capable of reasoning multimodal sarcastic tweets with word-level interpretation. Experimental results on a public dataset show that our model achieves the state-of-the-art performance compared with the existing models.
This work is supported in part by the Ministry of Science and Technology of China under Grants #2016QY02D0305 and #2018ZX10201001, and National Natural Science Foundation of China under Grants #11832001, #71702181 and #71621002.
Silvio Amir, Byron C Wallace, Hao Lyu, Paula Carvalho, and Mario J Silva. 2016. Modelling context with user embeddings for sarcasm detection in social media. In Proceedings of CoNLL, pages 167-177.
David Bamman and Noah A Smith. 2015. Contextualized sarcasm detection on twitter. In Proceedings of ICWSM, pages 574-577.
Christos Baziotis, Athanasiou Nikolaos, Pinelopi Papalampidi, Athanasia Kolovou, Georgios Paraskevopoulos, Nikolaos Ellinas, and Alexandros Potamianos. 2018. Ntua-slp at semeval-2018 task 3: Tracking ironic tweets using ensembles of word and character level attentive rnns. In Proceedings of SemEval, pages 613-621.
Yitao Cai, Huiyu Cai, and Xiaojun Wan. 2019. Multi-modal sarcasm detection in twitter with hierarchical fusion model. In Proceedings of ACL, pages 2506-2515.
Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019. Towards multimodal sarcasm detection (an obviously perfect paper). In Proceedings of ACL, pages 4619-4629.
Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcastic sentences in twitter and amazon. In Proceedings of CoNLL, pages 107-116.
Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In Proceedings of EMNLP, pages 1615-1625.
Aniruddha Ghosh and Tony Veale. 2017. Magnets for sarcasm: Making sarcasm detection timely, contextual and very personal. In Proceedings of EMNLP, pages 482-491.
Roberto González-Ibanez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in twitter: a closer look. In Proceedings of ACL, pages 581-586.
Alex Graves and Jürgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural networks, 18(5-6):602-610.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of CVPR, pages 770-778.
Irazú Hernández-Farías, José-Miguel Benedí, and Paolo Rosso. 2015. Applying basic features from sentiment analysis for automatic irony detection. In Proceedings of IbPRIA, pages 337-344.
Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. In Proceedings of ACL-IJCNLP, pages 757-762.
Aditya Joshi, Vaibhav Tripathi, Kevin Patel, Pushpak Bhattacharyya, and Mark Carman. 2016. Are word embedding-based features useful for sarcasm detection? In Proceedings of EMNLP, pages 1006-1011.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of EMNLP, pages 1746-1751.
Abhijit Mishra, Kuntal Dey, and Pushpak Bhattacharyya. 2017. Learning cognitive features from gaze data for sentiment and sarcasm classification using convolutional neural network. In Proceedings of ACL, pages 377-387.
Lotem Peled and Roi Reichart. 2017. Sarcasm sign: Interpreting sarcasm with sentiment based monolingual machine translation. In Proceedings of ACL, pages 1690-1700.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of EMNLP, pages 1532-1543.
Soujanya Poria, Erik Cambria, Devamanyu Hazarika, and Prateek Vij. 2016. A deeper look into sarcastic tweets using deep convolutional neural networks. In Proceedings of COLING, pages 1601-1612.
Ashwin Rajadesingan, Reza Zafarani, and Huan Liu. 2015. Sarcasm detection on twitter: A behavioral modeling approach. In Proceedings of WSDM, pages 97-106.
Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of EMNLP, pages 704-714.
Rossano Schifanella, Paloma de Juan, Joel Tetreault, and Liangliang Cao. 2016. Detecting sarcasm in multimodal social platforms. In Proceedings of ACM MM, pages 1136-1145.
Yi Tay, Anh Tuan Luu, Siu Cheung Hui, and Jian Su. 2018. Reasoning with sarcasm by reading in between. In Proceedings of ACL, pages 1010-1020.
Oren Tsur, Dmitry Davidov, and Ari Rappoport. 2010. *IcWsm—a great catchy name: Semi-supervised recognition of sarcastic sentences in online product reviews*. In *Proceedings of ICWSM*, pages 162-169.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS, pages 5998-6008.
3785
Penghui Wei, Wenji Mao, and Chen Guandan. 2019. A topic-aware reinforced model for weakly supervised stance detection. In Proceedings of AAAI, pages 7249-7256.
Meishan Zhang, Yue Zhang, and Guohong Fu. 2016. Tweet sarcasm detection using deep neural network. In Proceedings of COLING, pages 2449-2460.