SlowGuess/ACL_2020_OCR huggingface dataset:
Patna, Bihar, India-801106
{1821CS17, dhanush.cs16, asif, pb}@iitp.ac.in
In this paper, we hypothesize that sarcasm is closely related to sentiment and emotion, and thereby propose a multi-task deep learning framework to solve all these three problems simultaneously in a multi-modal conversational scenario. We, at first, manually annotate the recently released multi-modal MUStARD sarcasm dataset with sentiment and emotion classes, both implicit and explicit. For multitasking, we propose two attention mechanisms, viz. Inter-segment Inter-modal Attention ( $I_{e}$ -Attention) and Intra-segment Inter-modal Attention ( $I_{a}$ -Attention). The main motivation of  $I_{e}$ -Attention is to learn the relationship between the different segments of the sentence across the modalities. In contrast,  $I_{a}$ -Attention focuses within the same segment of the sentence across the modalities. Finally, representations from both the attentions are concatenated and shared across the five classes (i.e., sarcasm, implicit sentiment, explicit sentiment, implicit emotion, explicit emotion) for multi-tasking. Experimental results on the extended version of the MUStARD dataset show the efficacy of our proposed approach for sarcasm detection over the existing state-of-the-art systems. The evaluation also shows that the proposed multi-task framework yields better performance for the primary task, i.e., sarcasm detection, with the help of two secondary tasks, emotion and sentiment analysis.
Sarcasm is an essential aspect of daily conversation, and it adds more fun to the language. Oscar Wilde, an Irish poet-playwright, quotes, "Sarcasm is the lowest form of wit, but the highest form of intelligence". Irrespective of its relation with intelligence, sarcasm is often challenging to understand.
Sarcasm is often used to convey thinly veiled disapproval humorously. This can be easily depicted through the following example, "This is so
good, that I am gonna enjoy it in the balcony. I can enjoy my view, whilst I enjoy my dessert." This utterance, at an outer glance, conveys that the speaker is extremely pleased with his dessert and wants to elevate the experience by enjoying it in the balcony. But, careful observation of the sentiment and emotion of the speaker helps us understand that the speaker is disgusted with the dessert and has a negative sentiment during the utterance (c.f. Figure 1). This is where sentiment and emotion come into the picture. Sentiment, emotion and sarcasm are highly intertwined, and one helps in the understanding of the others better.
Figure 1: Example to show that sentiment and emotion of the speaker can influence sarcasm detection
Even though sentiment, emotion, and sarcasm are related, sarcasm was treated separately from its other counterparts in the past due to its complexity and its high dependency on the context. Moreover, multi-modal input helps the model to understand the intent and the sentiment of the speaker with more certainty. Thus in the context of a dialogue, multi-modal data such as video (acoustic + visual) along with text helps to understand the sentiment and emotion of the speaker, and in turn, helps to detect sarcasm in the conversation.
In this paper, we exploit these relationships, and make use of sentiment and emotion of the speaker for predicting sarcasm, specifically for the task, in a multi-modal conversational context. The main contributions and/or attributes of our proposed research are as follows: (a). we propose a multi-task
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4351-4360 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
learning framework for multi-modal sarcasm, sentiment, and emotion analysis. We leverage the utility of sentiment and emotion of the speaker to predict sarcasm. In our multi-task framework, sarcasm is treated as the primary task, whereas emotion analysis and sentiment analysis are considered as the secondary tasks. (b). We also propose two attention mechanisms viz.  $I_{e}$ -Attention and  $I_{a}$ -Attention to better combine the information across the modalities to effectively classify sarcasm, sentiment, and emotion. (c). We annotate the recently released Sarcasm dataset, MUStARD with sentiment and emotion classes (both implicit and explicit), and (d). We present the state-of-the-art for sarcasm prediction in multi-modal scenario.
A survey of the literature suggests that a multimodal approach towards sarcasm detection is a fairly new approach rather than a text-based classification. Traditionally, rule-based classification (Joshi et al., 2017; Veale and Hao, 2010) approaches were used for sarcasm detection. Poria et al. (2016) have exploited sentiment and emotion features extracted from the pre-trained models for sentiment, emotion, and personality on a text corpus, and use them to predict sarcasm through a Convolutional Neural Network.
In recent times, the use of multi-modal sources of information has gained significant attention to the researchers for affective computing. Mai et al. (2019) proposed a new two-level strategy (Divide, Conquer, and Combine) for feature fusion through a Hierarchical Feature Fusion Network for multimodal affective computing. Chauhan et al. (2019) exploits the interaction between a pair of modalities through an application of Inter-modal Interaction Module (IIM) that closely follows the concepts of an auto-encoder for the multi-modal sentiment and emotion analysis. Ghosal et al. (2018) proposed a contextual inter-modal attention based framework for multi-modal sentiment classification. In other work (Akhtar et al., 2019), an attention-based multitask learning framework has been introduced for sentiment and emotion recognition.
Although multi-modal sources of information (e.g., audio, visual, along with text) offers more evidence in detecting sarcasm, this has not been attempted much, one of the main reasons being the non-availability of multi-modal datasets. Recently, researchers (Castro et al., 2019) have started
exploiting multi-modal sources of information for sarcasm detection. It is true that the modalities like acoustic and visual often provide more evidences about the context of the utterance in comparison to text. For sarcasm detection, the very first multimodal dataset named as MUStARD has been very recently released by Castro et al. (2019), where the authors used a Support Vector Machine (SVM) classifier for sarcasm detection.
In our current work, we at first extend the MUs-tARD dataset (Castro et al., 2019) by manually labeling each utterance with sentiment and emotion labels. Thereafter, we propose a deep learning based approach along with two attention mechanisms ( $I_{e}$ -Attention and  $I_{a}$ -Attention) to leverage the sentiment and emotion for predicting sarcasm in a multi-modal multi-task framework. Further, to the best of our knowledge, this is the very first attempt at solving the multi-modal sarcasm detection problem in a deep multi-task framework. We demonstrate through a detailed empirical evaluation that sarcasm detection can be improved significantly if we are successful in leveraging the knowledge of emotion and sentiment using an effective multi-task framework.
The MUStARD (Castro et al., 2019) dataset consists of conversational audio-visual utterances (total of 3.68 hours in length). This dataset consists of 690 samples, and each sample consists of utterance accompanied by its context and sarcasm label. The samples were collected from 4 popular TV Series viz., Friends, The Big Bang Theory, The Golden Girls, and Sarcasmaholics Anonymous and manually annotated for the sarcasm label. The dataset is balanced with an equal number of samples for both sarcastic and non-sarcastic labels. The utterance in each sample consists of a single sentence, while the context associated with it consists of multiple sentences that precede the corresponding utterance in the dialogue. We manually re-annotated this dataset to introduce sentiment and emotion labels in addition to sarcasm. We define two kinds of emotion and sentiment values viz., implicit and explicit, which are discussed in the following subsections.
3.1 Sentiment
For sentiment annotation of an utterance, we consider both implicit and explicit affect information. The implicit sentiment of an utterance is determined
with the help of context. Whereas, explicit sentiment of an utterance is determined directly from itself, and no external knowledge from the context is required to infer it. We consider three sentiment classes, namely positive, negative and neutral. For the example in Figure 1, the implicit sentiment would be Negative, whereas explicit sentiment is Positive.
Table 1 shows the overall ratio of implicit and explicit sentiment labels, respectively. Whereas, Figure 2a and Figure 2b depict the show-wise ratio and distribution of each label.
Implicit Sentiment	Explicit Sentiment
Neg	Neu	Pos	Neg	Neu	Pos
391	89	210	246	119	325
(a) Show-wise-IS.
(b) Show-wise-ES.
Figure 2: Distribution of implicit sentiment (IS) and explicit sentiment (ES).
3.2 Emotion
Like sentiment, we annotate each sentence on the context and utterance for the implicit and explicit emotion. We annotate the dataset for 9 emotion values, viz. anger (An), excited (Ex), fear (Fr), sad (Sd), surprised (Sp), frustrated (Fs), happy (Hp), neutral (Neu) and disgust (Dg). Each utterance and context sentence are annotated, and these can have multiple labels per sentence for both implicit and explicit emotion. In the example of Figure 1, the implicit emotion of the speaker would be disgust while the explicit emotion is happy.
Table 2 shows the overall ratio of implicit and explicit emotion labels, respectively. Whereas Figure 3a and Figure 3b depict the show-wise ratio and distribution of each label.
3.3 Annotation Guidelines
We annotate all the samples with four labels (implicit sentiment/emotion and explicit sentiment/emotion). We employ three graduate students highly proficient in the English language with prior
Table 1: Sentiment distribution.
Explicit Emotion
An	Ex	Fr	Sd	Sp	Fs	Hp	Neu	Dg
54	30	6	118	35	23	206	228	10
Implicit Emotion
An	Ex	Fr	Sd	Sp	Fs	Hp	Neu	Dg
97	18	14	121	29	57	143	198	39
Table 2: Emotion distribution.
(a) Show-wise-IE.
(b) Show-wise-EE.
Figure 3: Distribution of implicit emotion (IE) and explicit emotion (EE).
experience in labeling sentiment, emotion, and sarcasm. The guidelines for annotation, along with some examples, were explained to the annotators before starting the annotation process.
The annotators were asked to annotate every utterance with as many emotions present in the utterance as possible, along with the sentiment. Initially, the dataset was annotated for explicit labels, with only the utterances provided to the annotators. Later, for the implicit labels, we also made the corresponding context video available to provide the relevant information for each sample. This method helps the annotators to resolve the ambiguity between the implicit and explicit labels. A majority voting scheme was used for selecting the final emotion and sentiment. We achieve an overall Fleiss' (Fleiss, 1971) kappa score of 0.81, which is considered to be reliable.
In this section, we describe our proposed methodology, where we aim to leverage the multi-modal sentiment and emotion information for solving the problem of multi-modal sarcasm detection in a multi-task framework. We propose a segment-wise inter-modal attention based framework for our task. We depict the overall architecture in Figure 4. The extended dataset with annotation guidelines and source code are available at http://www.iitp.ac.in/~ai-nlp-ml/resources.html.
Each sample in the dataset consists of an utterance  $(u)$  accompanied by its context  $(c)$  and labels
Figure 4: Overall architecture of the proposed multi-modal sarcasm detection framework.
(sarcasm, implicit sentiment, explicit sentiment, implicit emotion, and explicit emotion). The context associated with the utterance consists of multiple sentences (say,  $N$ ) that precede the corresponding utterance in the dialogue. Each utterance and its' context is associated with its' speaker i.e., speaker of utterance  $(SP_{u})$  and speaker of context  $(SP_{c})$ , respectively. We represent  $SP_{u}$  and  $SP_{c}$  by using a one-hot vector embedding.
We divide our proposed methodology into three subsections i.e., Input Layer, Attention Mechanism and Output Layer, which are described below:
4.1 Input Layer
The proposed model takes multi-modal inputs i.e., text  $(T)$ , acoustic  $(A)$ , and visual  $(V)$ . We describe the utterance and its' context for all the modalities below:
4.1.1 Text
Utterance: Let us assume, in an utterance, there  $n_t$  number of words  $w_{1:n_t} = w_1, \dots, w_{n_t}$ , where  $w_j \in \mathbb{R}^{d_t}$ ,  $d_t = 300$ , and  $w_j$ s are obtained using fastText word embeddings (Joulin et al., 2016). The utterance is then passed through a bidirectional Gated Recurrent Unit (Cho et al., 2014)  $(BiGRU_T^1)$  to learn the contextual relationship between the words. We apply the attention over the output of  $BiGRU_T$  to extract the important contributing words w.r.t. sarcasm. Finally, we apply  $BiGRU_F^2$  to extract the sentence level features. We then concatenate the speaker information of the
utterance with the output of  $BiGRU_{F}$ . This is denoted by  $T_{u} + SP_{u}$ , where  $T_{u}$  denotes the utterance for the text modality and  $SP_{u}$  denotes the speaker for that particular utterance.
Context: There are  $N_{c}$  number of sentences in the context where each sentence has  $n_{tc}$  words. For each sentence, words are passed through  $BiGRU_{F}$  to learn the contextual relationship between the words, and to obtain the sentence-wise representation. Then, we apply self-attention over the output of  $BiGRU_{F}$  to extract the important contributing sentences for the utterance. Finally, we concatenate the speaker information with each sentence and pass through the  $BiGRU_{F}$  to obtain the  $T_{c} + SP_{c}$ , where  $T_{c}$  denotes the context of the text modality, and  $SP_{c}$  denotes the speaker of that context.
4.1.2 Visual
Utterance: Let us assume there are  $n_v$  number of visual frames w.r.t. an utterance. We take the average of all frames to extract the sentence level information for the visual modality (Castro et al., 2019), and concatenate this with the speaker information. This is denoted as  $V_u + SP_u$ , where  $V_u \in \mathbb{R}^{d_v}$  and  $d_v = 2048$ .
Context: Given  $n_{vc}$  number of visual frames w.r.t. all the sentences, we take the average of all the visual frames (Castro et al., 2019) to extract the context level information, and denote this as  $V_{c}$ . As sentence-wise visual frames are not provided in the dataset, speaker information is not considered.
4.1.3 Acoustic
Utterance: Given  $n_a$  number of frames for the acoustic w.r.t. an utterance, we take the average of all the frames to extract the sentence level in
$^{1}BiGRU_{T}$  refers to the Bi-directional GRU units where output from all the time steps are forwarded in the model.  $^{2}BiGRU_{F}$  refers to the Bi-directional GRU units where output from the last time step is forwarded in the model.
formation (Castro et al., 2019), and concatenate with the speaker of the utterance. We denote this as  $A_{u} + SP_{u}$ , where  $A_{u} \in \mathbb{R}^{d_{a}}$  and  $d_{a} = 283$  corresponds to the utterance of the acoustic modality.
Context: For text, we concatenate the utterance  $(T_{u} + SP_{u})$  with its context  $(T_{c} + SP_{c})$ . For visual, we concatenate the utterance  $(V_{u} + SP_{u})$  with its context  $(V_{c})$  while for acoustic, we consider only the utterance  $A_{u} + SP_{u}$  (c.f. Figure 4). We do not consider any context information of the acoustics as it often contains information of many speakers, background noise, and noise due to laughter cues (which is not a part of the conversation). Hence, it might be difficult to disambiguate this with the laughter part of the conversation. Whereas, in the case of visual modality, it majorly contains the image of the speaker along with sentiment and emotion information. Thus, visual will not have a similar kind of problem as acoustic.
It is also to be noted that for a fair comparison with the state-of-the-art system (Castro et al., 2019), we take the average of the acoustic and visual features across the sentences.
4.2 Attention Mechanism
In any multi-modal information analysis, it is crucial to identify the important feature segments from each modality, so that when these are combined together can improve the overall performance. Here, we propose two attention mechanisms: (i). Inter-segment Inter-modal Attention  $(I_{e}$  -Attention), and (ii). Intra-segment Inter-modal Attention  $(I_{a}$  -Attention).
First, we pass the input representation from all the three modalities through a fully-connected layer  $(Dense_{d})$  to obtain the feature vector of length (d). These feature vectors are then forwarded to the aforementioned attention mechanisms.
4.2.1 Inter-segment Inter-modal Attention
For each modality, we first split the feature vector into k-segments to extract the fine level information. We aim to learn the relationship between the feature vector of a segment of an utterance in one modality and feature vector of the another segment of the same utterance in another modality through this mechanism (c.f. Figure 5). Then, an  $I_{e}$ -Attention is applied among the segments for every possible pair of modalities viz., TV, VT, TA, AT, AV, and VA. The overall procedure of  $I_{e}$ -Attention is depicted in Algorithm 1.
Figure 5: Procedure of the proposed  $I_{e}$ -Attention Mechanism.
4.2.2 Intra-segment Inter-modal Attention
For each utterance, we first concatenate the feature vectors (i.e.,  $\in \mathbb{R}^d$ ) obtained from the three modalities i.e.,  $\in \mathbb{R}^{3\times d}$  (c.f. Figure 6) and then split the feature vector into k-segments (i.e.,  $\in \mathbb{R}^{3\times \frac{d}{k}}$ ). Now, we have a mixed representation of all the modalities, i.e. visual, audio and text. The aim is, for a specific segment of any particular utterance, to establish the relationship between the feature vectors obtained from the different modalities.
Figure 6: Procedure of the proposed  $I_{a}$ -Attention mechanism
Motivated by the residual skip connection (He et al., 2016), the outputs of  $I_{e}$ -Attention and  $I_{a}$ -Attention along with the representations of individual modalities are concatenated (c.f Figure 4). Finally, the concatenated representation is shared across the five branches of our proposed network (i.e., sarcasm, I-sentiment, E-sentiment, I-emotion, & E-emotion) corresponding to three tasks, classification for the prediction (one for each task in the multi-task framework). Sarcasm and sentiment branches contain a Softmax layer for the final classification, while the emotion branch contains a Sigmoid layer for the classification. The shared representation will receive gradients of error from the five branches (sarcasm, I-sentiment, E-sentiment, I-emotion, & E-emotion), and accordingly adjusts the weights of the models. Thus, the shared representations will not be biased to any particular task, and it will assist the model in achieving better generalization for the multiple tasks.
We divide the whole process into four categories: i). utterance without context without speaker (i.e., we do not use the information of context and its' speaker with utterance); ii). utterance with context without speaker (i.e., we use the context information with utterance but not speaker information); iii). utterance without context with speaker (i.e., we use the speaker information with utterance but not context information); and iv). utterance with context with speaker (i.e., we use the context and its' speaker information with utterance).
We perform all the experiments for the setup utterances without context and speaker information (case i). Hence, even though the sentiment and emotion labels were annotated for both the context and utterance, we use the labels associated with utterances only for our experiments.
Our experimental setup is mainly divided into two main parts (Castro et al., 2019):
- Speaker Independent Setup: In this experiment, samples from The Big Bang Theory, The Golden Girls, and Sarcasmaholics Anonymous were considered for the training, and samples from the Friends Series were considered as the test set. Following this step, we
were able to reduce the effect of the speaker in the model.
- Speaker Dependent Setup: This setup corresponds to the five-fold cross-validation experiments, where each fold contains samples taken randomly in a stratified manner from all the series.
We evaluate our proposed model on the multimodal sarcasm dataset<sup>3</sup>, which we extended by incorporating both emotion and sentiment values. We perform grid search to find the optimal hyperparameters (c.f. Table 3). Though we aim for a generic hyper-parameter configuration for all the experiments, in some cases, a different choice of the parameter has a significant effect. Therefore, we choose different parameters for a different set of experiments.
Parameters	Speaker Dependent	Speaker Independent
Bi-GRU	2×200 neurons, dropout=0.3
Dense layer	200 neurons, dropout=0.3
Activations	ReLU
Optimizer	Adam (lr=0.001)
Output	Softmax (Sent) & Sigmoid (Emo)
Loss	Categorical cross-entropy (Sent)
Binary cross-entropy (Emo)
Batch	32
Epochs	200
#Segments (k)	50	25
Table 3: Model configurations
We implement our proposed model on the Python-based Keras deep learning library. As the evaluation metric, we employ precision (P), recall (R), and F1-score (F1) for sarcasm detection. We use Adam as an optimizer, Softmax as a classifier for sarcasm and sentiment classification, and the categorical cross-entropy as a loss function. For emotion recognition, we use Sigmoid as an activation function and optimize the binary cross-entropy as the loss.
We evaluate our proposed architecture with all the possible input combinations i.e. bi-modal  $(T + V, T + A, A + V)$  and tri-modal  $(T + V + A)$ . We do not consider uni-modal inputs  $(T, A, V)$  because our proposed attention mechanism requires at least two modalities. We show the obtained results in Table 4, that outlines the comparison between the multi-task (MTL) and single-task (STL) learning frameworks
<sup>3</sup>https://github.com/soujanyaporia/MUSTARD
Labels	T + V	T + A	A + V	T + A + V
P	R	F1	P	R	F1	P	R	F1	P	R	F1
Speaker Dependent	STL	Sar	71.52	70.61	69.32	64.20	64.20	63.88	71.90	71.01	70.64	72.08	71.62	72.01
MTL	Sar + Sent	69.65	69.42	69.33	64.09	60.72	58.21	72.20	71.45	71.18	72.52	71.73	72.07
Sar + Emo	71.76	70.86	70.54	65.76	65.65	65.60	72.60	71.59	71.25	72.76	71.88	72.11
Sar + Sent + Emo	72.76	71.88	71.61	62.23	61.15	59.61	72.73	71.88	71.81	73.40	72.75	72.57
Speaker Independent	STL	Sar	60.11	60.18	60.16	58.23	57.69	57.91	60.44	60.96	60.52	65.98	65.45	65.60
MTL	Sar + Sent	62.74	62.92	62.81	59.25	59.55	52.89	61.60	60.95	61.14	66.97	63.76	63.68
Sar + Emo	65.11	65.16	65.13	59.59	59.55	59.58	63.19	63.76	62.91	66.35	65.44	65.63
Sar + Sent + Emo	65.48	65.48	65.67	59.13	59.98	50.27	65.59	63.76	63.90	69.53	66.01	65.90
without taking context and speaker information into consideration. We observe that Tri-modal  $(T + A + V)$  shows better performance over the bi-modal setups.
For  $STL$ , experiments with only sarcasm class are used, whereas for  $MTL$ , we use three sets of experiments, i.e. sarcasm with sentiment  $(Sar + Sent)$ , sarcasm with emotion  $(Sar + Emo)$ , and sarcasm with sentiment and emotion  $(Sar + Sent + Emo)$ . For sarcasm classification, we observe that multitask learning with sentiment and emotion together shows better performance for both the setups (i.e. speaker dependent and speaker independent) over the single-task learning framework. It is evident from the empirical evaluation, that both sentiment and emotion assist sarcasm through the sharing of knowledge, and hence  $MTL$  framework yields better prediction compared to the  $STL$  framework (c.f. Table 4).