SlowGuess/ACL_2020_OCR huggingface dataset:
We also show the results for the single-task  $(T + A + V)$  experiments under speaker-dependent and speaker-independent setups for sentiment and emotion. These results can be considered as baseline for the same. The detailed description of sentiment and emotion are described in Section 3.1 and Section 3.2, respectively.
For Sentiment Analysis, the results are shown in Table 5.
Table 4: Single Task vs Multi Task: Without Context and Without Speaker information.
Speaker Dependent
Implicit Sentiment	Explicit Sentiment
P	R	F1	P	R	F1
49.27	57.39	49.12	48.32	52.46	48.11
Speaker Independent
P	R	F1	P	R	F1
47.05	49.15	40.99	47.73	50.0	45.24
Similarly, for emotion analysis, the results are shown in Table 6. Along with it, results from the single-Task experiments for each emotion under implicit emotion and explicit emotion for Speaker Dependent and Speaker Independent setups are shown
in Table 7 and Table 8, respectively. As each utterance can have multiple emotion labels, we take all the emotions whose respective values are above a threshold. We optimize and cross-validate the evaluation metrics and set the threshold as 0.5 0.45 for speaker-dependent and speaker-independent setups, respectively.
Table 5: Results for Single-task experiments for Sentiment analysis  $(T + A + V)$
Speaker Dependent
Implicit Sentiment	Explicit Sentiment
P	R	F1	P	R	F1
80.66	88.51	83.57	85.01	88.90	85.12
Speaker Independent
P	R	F1	P	R	F1
81.77	88.29	83.88	83.64	88.35	84.37
Table 6: Results for Single-task experiments for Emotion analysis  $(T + A + V)$ .
Speaker Dependent
Setup	Implicit Emotion	Explicit Emotion
P	R	F1	P	R	F1
An	74.0	85.9	79.5	85.0	92.2	88.4
Ex	94.9	97.3	96.1	91.5	95.6	93.5
Fr	95.9	97.8	96.9	98.3	99.1	98.7
Sd	68.0	82.3	74.5	72.1	83.0	75.5
Sp	91.8	95.8	93.7	90.1	94.9	92.5
Fs	84.2	91.7	87.8	93.4	96.7	95.0
Hp	67.1	79.5	71.4	66.6	71.7	66.5
Neu	60.9	71.6	60.5	70.9	68.3	58.1
Dg	89.0	94.3	91.6	97.1	98.5	97.8
Table 7: Emotion-wise results for Single-Task experiments - Speaker Dependent setup.
Speaker Independent
Setup	Implicit Emotion	Explicit Emotion
P	R	F1	P	R	F1
An	72.0	84.8	77.9	81.3	90.1	85.5
Ex	95.6	97.7	96.6	94.5	97.2	95.8
Fr	95.0	97.5	96.2	97.8	98.9	98.3
Sd	74.8	86.5	80.3	72.9	85.4	78.7
Sp	92.8	96.3	94.5	93.4	96.6	94.9
Fs	88.5	94.1	91.2	91.7	95.8	93.7
Hp	65.6	71.6	60.8	67.4	62.9	49.6
Neu	50.9	71.3	59.4	52.9	72.7	61.3
Dg	94.5	97.2	95.8	96.1	98.0	97.0
Table 8: Emotion-wise results for Single-Task experiments - Speaker Independent setup.
We further evaluate our proposed model by incorporating context and speaker information to form the three combinations of experiments viz. With Context Without Speaker, Without Context With Speaker, With Context and Speaker (c.f. Table 9). The experiments without context and without speaker information are same as the tri-modal setup in Table 4. The maximum improvement  $(1 - 5\% \uparrow)$  in performance is observed when the speaker information alone is incorporated in the tri-modal setup. Whereas in Speaker Independent Setup, incorporating both context and speaker information significantly improves the performance  $(1 - 5\% \uparrow)$ .
Setup	Speaker Dependent	Speaker Independent
Context	Speaker	P	R	F1	P	R	F1
X	X	73.40	72.75	72.57	69.53	66.01	65.90
X	✓	77.09	76.67	76.57	74.69	74.43	74.51
✓	X	72.34	71.88	71.74	71.51	71.35	70.46
✓	✓	76.07	75.79	75.72	74.88	75.01	74.72
To understand the contribution of  $I_{e}$ -Attention and  $I_{a}$ -Attention towards the performance of the model, an ablation study was performed without the attention-mechanisms (c.f. Table 10).
Table 9: Results for different combination of Context-Speaker Experiments
Setup	Speaker Dependent	Speaker Independent
P	R	F1	P	R	F1
W/o Attention	71.53	69.71	69.02	60.53	61.23	60.44
Proposed	73.40	72.75	72.57	69.53	66.01	65.90
Table 10: Ablation study: Proposed Attention v/s Without Attention.
5.3 Comparative Analysis
We compare, under the similar experimental setups, the results obtained in our proposed model (without context and speaker) against the existing models called as baseline (Castro et al., 2019), which also made use of the same dataset. The comparative analysis is shown in Table 11. For tri-modal experiments, our proposed multi-modal multi-task framework achieves the best precision of  $73.40\%$ $(1.5\% \uparrow)$ , recall of  $72.75\%$ $(1.4\% \uparrow)$  and F1-score of  $72.57\%$ $(1.1\% \uparrow)$  for the proposed multi-task model (Sar + Sent + Emo) as compared to precision of  $71.9\%$ , recall of  $71.4\%$ , F1-score of  $71.5\%$  of the state-of-the-art system. We observe that both sentiment and emotion help in improving the efficiency of sarcasm detection. Similarly, for the Speaker Independent setup, we obtain an improvement of  $5.2\%$  in precision,  $3.4\%$  in recall, and  $3.1\%$  in F1-score.
We perform statistical significance test (paired  $T$ -test) on the obtained results and observe that performance improvement in the proposed model over the state-of-the-art is significant with  $95\%$  confidence (i.e.  $p$ -value  $< 0.05$ ).
We analyze the attention weights to understand the learning behavior of the proposed framework. We take an utterance i.e., "I love that you take pride in your looks, even when I have to pee in the morning, and you're in there spending an hour on your hair." (c.f Table 12) from the dataset which is a sarcastic utterance. The MTL  $(\mathrm{Sar} + \mathrm{Sent} + \mathrm{Emo})$  correctly classifies this utterance as sarcastic, while the STL (Sar) predicts it as non-sarcastic. In this utterance, we feel that the speaker is pleased and happy (explicit emotion) where he is angry (implicit emotion) on the other person and is expressing that anger sarcastically.
(a)  $TV$
(b)  $AV$
(c)  $TA$
(d)  $VT$
(e)  $VA$
(f)  $AT$
Figure 7: Heatmaps for the all combinations of modalities for  $I_{e}$ -Attention.
We analyze the heatmaps of the attention weights  $(I_{e}$  -Attention and  $I_{a}$  -Attention) for the above utterance. Each cell of heatmaps for  $I_{e}$  -Attention (c.f. Figure 7) represents the different segments of the sentence across the modalities. Cell  $(i,j)$  of the heatmap for the modalities (say,  $TV$  ) represents the influence of  $s_j$  of visual on  $s_i$  of textual modality, in predicting the output (where  $s_i$  represents  $i^{th}$  segment of the feature vector from the respective modality). In Figure 7a, for the first segment of the utterance (i.e.,  $s_1$  ) of the textual modality, the model puts more attention weights to the different segments of the utterance (i.e.,  $s_6,$ $s_7,s_9$  ,and  $s_{10}$  ) of visual modality to classify the
Setup	Model	T + V	T + A	A + V	T + A + V
P	R	F1	P	R	F1	P	R	F1	P	R	F1
Speaker Dependent	Baseline	72.0	71.6	71.6	66.6	66.2	66.2	66.2	65.7	65.7	71.9	71.4	71.5
Proposed Model	72.8	71.9	71.6	62.2	61.2	59.6	72.7	71.9	71.8	73.4	72.8	72.6
T-test	-	-	-	-	-	-	-	-	-	0.0023	0.0098	0.0056
Speaker Independent	Baseline	62.2	61.5	61.7	64.7	62.9	63.1	64.1	61.8	61.9	64.3	62.6	62.8
Proposed Model	65.5	65.5	65.7	59.1	60.0	50.3	65.6	63.8	63.9	69.5	66.0	65.9
T-test	-	-	-	-	-	-	-	-	-	0.0002	0.0006	0.0012
Table 11: Comparative Analysis of the proposed approach with recent state-of-the-art systems. We evaluated on extended, publicly available MUSTARD dataset (Castro et al., 2019). Significance test  $p$ -values  $< 0.05$
Utterances	Sarcasm (T+A+V)
Actual	STL	MTL
1	Oh yeah ok, including the waffles last week, you now owe me, seventeen zillion dollars.	S	NS	S
2	I love that you take pride in your looks, even when I have to pee in the morning, and you're in there spending an hour on your hair.	S	NS	S
3	Now?! - No, after my tongue has swollen to the size of a brisket!	S	S	S
4	There's no hurry. Tell them more about their secret love for each other.	NS	S	NS
5	I'm not saying that you're not fun. You're the most fun person I know.	NS	S	NS
6	Well, I'm sorry, too, but there's just no room for you in my wallet.	S	S	S
Table 12: Comparison between multi-task learning (Sar + Sent + Emo) and single-task learning (Sar) frameworks for tri-modal  $(\mathrm{T} + \mathrm{A} + \mathrm{V})$  inputs. Few error cases where MTL framework performs better than the STL framework.
utterance correctly. Similarly, for  $I_{a}$ -Attention, each  $\operatorname{cell}(i,j)$  of the heatmap (c.f. Figure 8)
signifies the influence of  $s_j$  on  $s_i$  in predicting the output (where  $s_i$  represents  $i^{th}$  segment of the concatenated feature vector from all modalities). We observe that for a particular segment of the utterance (say  $s_6$ ), the model puts more
weights to itself rather than the others.
We also observe that in the bi-modal  $(\mathrm{T} + \mathrm{A})$  experiment (c.f. Table 4) our model does not perform at par when we attempt to solve all the three tasks, i.e. sarcasm, sentiment, and emotion together. This may be attributed to the reason of not incorporating the visual information that contains rich affect cues in the forms of sentiment and emotion. Hence, the introduction of sentiment in the  $\mathrm{T} + \mathrm{A}$  setting might be confusing the model.
In this paper, we have proposed an effective deep learning-based multi-task model to simultaneously solve all the three problems, viz. sentiment analysis, emotion analysis and sarcasm detection. As there was no suitable labeled data available for this
problem, we have created the dataset by manually annotating an existing dataset of sarcasm with sentiment and emotion labels. We have introduced two attention mechanisms (i.e.,  $I_{e}$ -Attention and  $I_{a}$ -Attention), and incorporated the significance of context and speaker information w.r.t. sarcasm. Empirical evaluation results on the extended version of the MUStARD dataset suggests the efficacy of the proposed model for sarcasm analysis over the existing state-of-the-art systems. The evaluation also showed that the proposed multi-tasking framework achieves better performance for the primary task, i.e. sarcasm detection, with the help of emotion analysis and sentiment analysis, the two secondary tasks in our setting.
During our analysis, we found that the dataset is not big enough for a complex framework to learn from. Along with investigating new techniques, we hope that assembling a bigger curated dataset with quality annotations will help in better performance.
The research reported here is partially supported by SkyMap Global India Private Limited. Asif Ekbal acknowledges the Young Faculty Research Fellowship (YFRF), supported by Visvesvaraya PhD scheme for Electronics and IT, Ministry of Electronics and Information Technology (Meit/8Y), Government of India, being implemented by Digital India Corporation (formerly Media Lab Asia).
Md Shad Akhtar, Dushyant Singh Chauhan, Deepanway Ghosal, Soujanya Poria, Asif Ekbal, and Pushpak Bhattacharyya. 2019. Multi-task learning for multi-modal emotion recognition and sentiment analysis. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 370-379, Minneapolis, Minnesota. Association for Computational Linguistics.
Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019. Towards multimodal sarcasm detection (an _obviously_ perfect paper). arXiv preprint arXiv:1906.01815.
Dushyant Singh Chauhan, Md Shad Akhtar, Asif Ekbal, and Pushpak Bhattacharyya. 2019. Context-aware interactive attention for multi-modal sentiment and emotion analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5651-5661, Hong Kong, China. Association for Computational Linguistics.
KyungHyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. CoRR, abs/1409.1259.
Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many rater. *Psychological Bulletin*, 76:378-382.
Deepanway Ghosal, Md Shad Akhtar, Dushyant Singh Chauhan, Soujanya Poria, Asif Ekbal, and Pushpak Bhattacharyya. 2018. Contextual inter-modal attention for multi-modal sentiment analysis. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3454-3466, Brussels, Belgium. Association for Computational Linguistics.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
Aditya Joshi, Pushpak Bhattacharyya, and Mark J. Carman. 2017. Automatic sarcasm detection: A survey. ACM Comput. Surv., 50(5):73:1-73:22.
Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Herve Jégou, and Tomas Mikolov. 2016. Fasttext.zip: Compressing text classification models. arXiv preprint arXiv:1612.03651.
Sijie Mai, Haifeng Hu, and Songlong Xing. 2019. Divide, conquer and combine: Hierarchical feature fusion network with local and global perspectives for multimodal affective computing. In Proceedings of
the 57th Annual Meeting of the Association for Computational Linguistics, pages 481-492.
Soujanya Poria, Erik Cambria, Devamanyu Hazarika, and Prateek Vij. 2016. A deeper look into sarcastic tweets using deep convolutional neural networks. arXiv preprint arXiv:1610.08815.
Tony Veale and Yanfen Hao. 2010. Detecting ironic intent in creative comparisons. In *ECAI*, volume 215, pages 765-770.