SlowGuess/NAACL_2022_OCR huggingface dataset:
Yiyi Liu\*1,2, Yequan Wang\*3, Aixin Sun4, Xuying Meng5, Jing Li6, Jiafeng Guo1,2
$^{1}$ CAS Key Lab of Network Data Science and Technology, Institute of Computing Technology,
<sup>3</sup>Beijing Academy of Artificial Intelligence, Beijing, China
$^{4}$ School of Computer Science and Engineering, Nanyang Technological University, Singapore
$^{5}$ Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
<sup>6</sup>Inception Institute of Artificial Intelligence, Abu Dhabi, United Arab Emirates
tshwangyequan@gmail.com, axsun@ntu.edu.sg, jingli.phd@hotmail.com
{liuyiyil7s,mengxuying,guojiafeng}@ict.ac.cn
Sarcasm employs ambivalence, where one says something positive but actually means negative, and vice versa. The essence of sarcasm, which is also a sufficient and necessary condition, is the conflict between literal and implied sentiments expressed in one sentence. However, it is difficult to recognize such sentiment conflict because the sentiments are mixed or even implicit. As a result, the recognition of sophisticated and obscure sentiment brings in a great challenge to sarcasm detection. In this paper, we propose a Dual-Channel Framework by modeling both literal and implied sentiments separately. Based on this dual-channel framework, we design the Dual-Channel Network (DC-Net) to recognize sentiment conflict. Experiments on political debates (i.e., IAC-V1 and IAC-V2) and Twitter datasets show that our proposed DC-Net achieves state-of-the-art performance on sarcasm recognition. Our code is released to support research<sup>1</sup>.
Sarcasm is a complicated linguistic phenomenon. Intuitively, it means that one says something positive on surface form, while he/she actually expresses negative, vice versa (Liu, 2012; Morrison, 2008). Take the sentence "Final exam is the best gift on my birthday" as an example, the literal sentiment on surface is positive, which is reflected by the explicit sentiment words, i.e., "best gift". However, the factual part of the text (i.e., "final exam happens on birthday") implies that the sentiment expressed is negative. This example suggests
that it is the sentiment conflict that causes sarcasm linguistically.
However, modeling this linguistic nature of sarcasm is a great challenge due to the difficulty of digging sentiment conflict between the literal and the implied meanings. We know that non-sarcastic texts do not contain implied meaning, so the literal sentiment is consistent with the actual sentiment. But for sarcastic text, there is more than one meaning that coexists in one sentence. The literal meaning and the implied meaning are reflected in different sub-sentences. Even more challenging, sentiments behind the two meanings are mixed or even implicit.
Many existing studies adopt generic classification models for sarcasm recognition (Lou et al., 2021; Ghosh and Veale, 2016). However, these methods directly model the entire sentence without considering the contradictory meanings behind sarcastic texts. There are also studies using contrast patterns (e.g., phrase pair and word pair) as indicators to detect sarcasm, which is approaching the linguistic essence of sarcasm. Riloff et al. (2013); Joshi et al. (2015) detect contrast or incongruity patterns, i.e., the co-occurrence of positive sentiment phrases and negative situational phrases. Tay et al. (2018); Xiong et al. (2019) use attention mechanism to measure the sentiment conflict between word pairs in sarcastic texts. However, these methods emphasize too much on the explicit sentiment conflict on surface form (i.e., word/phrase level), which mainly reflect the literal meaning. As a result, the factual text is underestimated, which expresses the implied sentiment.
Dual-Channel Framework. In this paper, we propose a dual-channel framework to model the lit
*Indicates equal contribution
1https://github.com/yiyi-ict/
dual-channel-for-sarcasm
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1670 - 1680
Figure 1: The Dual-Channel Framework for sarcasm recognition.
eral sentiment and the implied sentiment simultaneously. This allows us to leverage the conflict between the two channels in a comprehensive way. Figure 1 depicts the proposed dual-channel framework. In this framework, literal channel and implied channel are used to detect the surface and the hidden meanings separately. Once sentiment conflict is detected, we could determine the existence of sarcasm. The design of our dual-channel framework balances the effect of literal and implied inputs and avoids focusing too much on either one channel while ignoring the other. Our framework covers existing sarcasm patterns, and could be further enhanced to detect more sentiment conflict patterns.
Based on this framework, we develop the Dual-Channel Network (DC-Net) to detect sarcasm. DC-Net contains four modules: decomposer, literal channel, implied channel, and analyzer. In general, sentiment words directly reflect the surface sentiment, while the text without sentiment words reflects the implied sentiment. Hence, we split the sentiment words of input text to literal channel, and the remaining words to implied channel by decomposer. Then we use the literal channel to model surface meaning, and the implied channel to model hidden meaning. Lastly, we use analyzer to recognize the conflict. Experiments on three benchmark datasets (i.e., IAC-V1, IAC-V2 and Tweets) show that our proposed DC-Net model achieves state-of-the-art performance.
The main contributions of this paper are twofold. First, to the best of our knowledge, the dual-channel framework is the first attempt to explicitly separate literal meaning and implied meaning to recognize sarcasm by detecting sentiment conflict. Second, experiments conducted on bench
mark datasets (i.e., IAC-V1/V2 and Tweets) show that DC-Net achieves state-of-the-art performance.
Prior methods of sarcasm recognition can be divided into traditional models and neural models. There are also methods considering context information, e.g., posting history (Hazarika et al., 2018; Zhang et al., 2016), and user profile (Poria et al., 2016; Kolchinski and Potts, 2018). However, such context may not be always available.
2.1 Traditional Models
Most traditional approaches adopt machine learning methods such as SVM with manually crafted rules or feature engineering. The features include sentiment lexicons (González-Ibanez et al., 2011; Patra et al., 2016), pragmatic features (i.e., emoticons (González-Ibanez et al., 2011), capitalization, punctuations (Joshi et al., 2015)), and pattern-based features (Riloff et al., 2013) et al.. Hee et al. (2018b) utilize common sense to assist sarcasm detection on Twitter. Accordingly, the accuracy of sarcasm recognition highly depends on the quality of features.
Rewriting key parts of a sentence manually is an expensive but effective method. Ghosh et al. (2015) believe that sarcasm involves a figurative meaning which is usually the opposite of literal meaning. They reframe sarcasm recognition as a literal/sarcastic word sense disambiguation problem. Then they paraphrase sarcastic texts manually to obtain target words that cause sarcastic disambiguation. This work is novel but heavily relies on manual paraphrasing and labeling of datasets to find target words. Moreover, target words are mostly limited to sentiment words. As a result, the model is dominated by these explicit sentiment words and ignores the implied channel.
Ghosh and Veale (2016) propose a model composed of CNN, LSTM and DNN to detect sarcasm. As attention mechanism has led to improvements in various NLP tasks, Tay et al. (2018); Xiong et al. (2019) use attention to capture the relationship of word pairs along with an LSTM to model the entire sentence. Lou et al. (2021) design a GCN-based model combining SenticNet (Cambria et al., 2020), dependency tree and LSTM with GCN (Kipf and Welling, 2017) together, which achieves promising
performance. Similar to previous studies, to better understand sarcasm, many approaches are able to utilize external information such as emoji expressions (Felbo et al., 2017), affective knowledge (Babanejad et al., 2020) and commonsense (Li et al., 2021). Joshi et al. (2017) provide a more comprehensive survey. Moreover, there have been many systems developed for a shared task (Ghosh et al., 2020). These models are rarely designed to reflect the essential features of the sarcasm phenomenon.
3 Dual-Channel Network (DC-Net)
The architecture of the proposed DC-Net is shown in Figure 2. It consists of four modules: decomposer, literal channel, implied channel, and analyzer. Given an input text, we use the decomposer to split it into two sub-sentences corresponding to the two channels. Then we use these two channels to derive literal and implied representations independently. Lastly, the analyzer predicts whether the text is sarcastic or not by detecting sentiment conflict.
3.1 Decomposer
The decomposer module is designed to split input text to the literal and implied channels. From numerous sarcastic corpora, we observe that sarcastic texts often contain evident sentiment words. More specifically, the literal channel itself is to reflect the intuitive sentiment. So it is reasonable to use sentiment lexicons as direct keywords. The remaining text expresses the implied sentiment. For example, sentiment words of input text (e.g., "best gift") represent positive, while the remaining part (e.g., "Final exam is on my birthday") implies the negative sentiment. Shown in Table 1, proportion of texts that contain sentiment words ranges from  $88\%$  to  $96\%$  in three datasets. Hence, using sentiment words to split input well serves the purpose.
Considering a text  $W_{T} = \{w_{1}, w_{2}, \ldots, w_{N}\}$  with  $N$  words, we decompose it into two pieces: the sentiment words  $W_{L}$ , and the remaining text  $W_{D}$  (see Figure 2).  $W_{L}$  is fed to the literal channel, and  $W_{D}$  to the implied channel. In this process, we use the sentiment lexicon released in Wilson et al. (2005) to pick up sentiment words. If no sentiment words are matched from the given text, the original text is used as the literal channel's input, which is the same as the implied channel. Note that in quite a few texts, sentiment words are adjectives or adverbs, deleting them from sentences has no much
Figure 2: The architecture of the DC-Net.
impact on the overall semantics. Although the text is not normative as expected after decomposing, we do not fill in the full text with placeholders.
3.2 Literal Channel
The literal channel includes an encoder, two linear layers, and a softmax classifier.  $\mathrm{Encoder}_L$  is used to encode the literal text  $W_{L}$  . Then we can get the literal representation  $v_{L}$  through
$$
v _ {L} = \operatorname {E n c o d e r} _ {L} \left(W _ {L}\right). \tag {1}
$$
Next, we use a softmax layer to compute the literal sentiment distribution based on the literal representation  $v_{L}$ .
$$
\mathcal {P} _ {l} = \operatorname {s o f t m a x} \left(W _ {r} v _ {L} + b _ {r}\right), \tag {2}
$$
where,  $W_{r}$  and  $b_{r}$  are parameters of the linear layer.
Considering the semantic complexity of sarcastic texts, a single representation of sentiment words may lose context information. So we adopt another  $\mathrm{Encoder}_T$  to encode the original text  $W_{T}$  and obtain the representation of the entire text  $v_{T}$  through
$$
v _ {T} = \operatorname {E n c o d e r} _ {T} \left(W _ {T}\right). \tag {3}
$$
Last, we concatenate the literal state  $v_{L}$  and the entire text's state  $v_{T}$ , followed by a linear layer and ReLU activation function to reduce dimension. Briefly, the final representation  $v_{L}^{\prime}$  of the literal channel could be formulated as:
$$
v _ {L} ^ {\prime} = \operatorname {R e L U} \left(W _ {l} \left[ v _ {L}; v _ {T} \right] + b _ {l}\right), \tag {4}
$$
where  $W_{l}$  and  $b_{l}$  are parameters of the second linear layer.
3.3 Implied Channel
In the implied channel, we also adopt an Encoder with the same structure but different parameters to encode the implied input text  $W_{D}$ , and the representation of the implied channel is formulated as:
$$
v _ {D} = \operatorname {E n c o d e r} _ {D} \left(W _ {D}\right). \tag {5}
$$
Similarly, we use softmax to calculate the implied sentiment distribution based on the implied hidden state  $v_{D}$ :
$$
\mathcal {P} _ {d} = \operatorname {s o f t m a x} \left(W _ {z} v _ {D} + b _ {z}\right), \tag {6}
$$
where  $W_{z}$  and  $b_{z}$  are parameters.
Again, we concatenate the implied hidden state  $v_{D}$  with the entire text's hidden state  $v_{T}$ , followed by a linear layer and an activation layer ReLU. The final representation  $v_{D}^{\prime}$  of the implied channel is formulated as:
$$
v _ {D} ^ {\prime} = \operatorname {R e L U} \left(W _ {d} \left[ v _ {D}; v _ {T} \right] + b _ {d}\right), \tag {7}
$$
where  $W_{d}$  and  $b_{d}$  are parameters.
Note that the structures of the two channels are symmetrical. However, the two encoders in the two channels do not share parameters, and their inputs are different. Since both channels are not specific to particular encoders, the dual-channel framework is able to adapt to mainstream encoders, e.g., LSTM (Hochreiter and Schmidhuber, 1997), CNN (Kim, 2014), Recursive Neural Network (Socher et al., 2011), BERT (Devlin et al., 2019) et al.. In DC-Net, we adopt Bi-LSTM as encoders for both channels.
3.4 Analyzer
The analyzer is designed to measure the conflict between the literal and the implied channels. We concatenate the literal representation  $v_{L}^{\prime}$  and the implied representation  $v_{D}^{\prime}$  and feed the result to a softmax layer. Other analyzers such as subtraction or cosine similarity also fit our design.
$$
\mathcal {P} _ {s} = \mathrm {s o f t m a x} (W _ {p} ([ v _ {L} ^ {\prime}, v _ {D} ^ {\prime} ]) + b _ {p}), \qquad (8)
$$
where  $W_{p}$  and  $b_{p}$  are parameters.
Although sarcasm has a strong correlation to literal sentiment and implied sentiment, we do not have gold labels for both sentiments. Hence, requesting the model to directly output sentiments on both channels may confuse the model. For this reason, we develop the objective function of sarcasm classification by adding objectives of the literal and implied channels.
3.5 Training Objective
The training objective of the proposed DC-Net model considers three aspects. One is to minimize the cross-entropy loss of the sarcasm probability distribution. The other two are to minimize the cross-entropy losses of the literal and that of the implied sentiment probability distributions respectively.
Sarcasm Objective. The sarcasm objective is to ensure the basic ability of detection. Hence, we use cross-entropy loss of sarcasm classification. The objective  $J_{s}$  is formulated as:
$$
J _ {s} (\theta) = \sum \text {c r o s s - e n t r o p y} \left(y _ {s}, \mathcal {P} _ {s}\right), \tag {9}
$$
where  $\mathcal{P}_s$  denotes the sarcasm probability distribution of the text. The groundtruth of the sarcasm label is  $y_{s}$
Literal Sentiment Objective. Due to the expensive manual annotations, we use sentiment words for approximate labeling, which is widely used in Eisenstein (2017); Taboada et al. (2011); Hu and Liu (2004). In our implementation, we determine the literal sentiment label based on the number of words with positive sentiment and the words with negative sentiment in input text. For sarcastic texts, if the number of positive words is greater than that of negative words, the literal sentiment label is positive and the implied sentiment label is negative, and vice versa. For non-sarcastic texts, both the literal sentiment label and the implied sentiment label are the same, determined by the number of positive/negative sentiment words.
The literal sentiment classification objective is then formulated as:
$$
J _ {l} (\theta) = \sum \text {c r o s s - e n t r o p y} \left(y _ {l}, \mathcal {P} _ {l}\right), \tag {10}
$$
where  $\mathcal{P}_l$  is the literal sentiment probability distribution. The label generated by the labeling processing of the literal sentiment is  $y_{l}$
Implied Sentiment Objective. We observe that literal sentiment and implied sentiment of sarcastic texts are often opposite. Using the implied labels based on the automatic labeling processing, we calculate the implied sentiment classification objective by
$$
J _ {d} (\theta) = \sum \text {c r o s s - e n t r o p y} \left(y _ {d}, \mathcal {P} _ {d}\right), \tag {11}
$$
where  $\mathcal{P}_d$  denotes the implied sentiment probability distribution. The label generated by the labeling processing of the implied sentiment is  $y_{d}$
Considering these three objectives, we obtain the final objective function  $L$  by adding them together:
$$
L (\theta) = \lambda_ {1} J _ {s} (\theta) + \lambda_ {2} J _ {l} (\theta) + \lambda_ {3} J _ {d} (\theta), \tag {12}
$$
where  $\theta$  is the parameter set of the model.  $\lambda_{1},\lambda_{2}$  and  $\lambda_{3}$  are used to leverage the contributions of the three objectives.
We conduct experiments on three benchmark datasets: IAC-V1, IAC-V2, and Tweets. These datasets do not contain context information such as historical tweet posts and user profiles. All of them have been widely used in evaluating sarcasm detection.
- IAC-V1 is collected from online political debates forum<sup>2</sup>. It is the subset of the Internet Argument Corpus (Lukin and Walker, 2013). The written language of IACs is English. Each instance, typically a sentence, is annotated with sarcasm label, either "sarcasm" or "non-sarcasm". Compared to tweets, texts of IAC are much longer and more normative.
- IAC-V2 (Oraby et al., 2016) contains more data than IAV-V1 (the two versions have a few overlaps). IAC-V2 divides sarcasm into three sub-types, (i.e., general sarcasm, hyperbole, and rhetorical questions). We use the largest subset (general sarcasm) in our experiments.
- Tweets dataset written in English is proposed in SemEval 2018 Task 3 Subtask A (Hee et al., 2018a). Each instance (i.e., a sentence) is labeled sarcastic or non-sarcastic. There are three variations of the text in this dataset: (i) original texts, (ii) texts with hashtags removed, and (iii) texts with hashtags and emoji expressions removed. Hashtags like "#not", "#sar-casm", and "#irony", are originally obtained from users. The hashtags are also used as prior knowledge for collecting sarcastic posts. In our experiments, we used the version without hashtags.
Table 1 reports the statistics. We observe that more than  $88\%$  of the texts contain sentiment
Table 1: Statistics of datasets. Avg  $\ell$  denotes the average length of texts in the number of tokens.  $s$  ratio is the proportion of texts that contain sentiment words.
Dataset	Train	Valid	Test	Avg ℓ	s ratio
IAC-V13	1,596	80	320	68	91%
IAC-V24	5,216	262	1,042	43	96%
Tweets5	3,634	200	784	14	88%
word(s). Hence, it is reasonable to decompose the original text into sentiment words and non-sentiment words, as inputs to the literal channel and implied channel, respectively. The number of instances in the three datasets is between  $1k$  and  $6k$ . All three datasets are class-balanced. The ratio of sarcastic instances and non-sarcastic instances is nearly 1:1. Due to the small size, the split of train-valid/test is important to avoid overfitting. For Tweets dataset, we follow the official train/test split. Then we randomly select  $5\%$  from training as valid sub-dataset. There is no official train-valid/test split for the two IAC datasets, so we split IAC datasets following the same ratio of Tweets. The baselines papers do not provide the split (or not conduct experiments on IAC datasets). So we cannot directly adopt the results of baselines reported in their original papers. Hence, we re-implement all baseline models on IAC-V1 and IAC-V2 datasets.
There are another three datasets for sarcasm detection. Riloff et al. (2013) and Ptacek et al. (2014) propose another two datasets based on Tweets, but they only provide tweet IDs. Due to modified authorization status, lots of tweets are unavailable or deleted. For this reason, we could not experiment on these two Tweet datasets. Khodak et al. (2018) build a large self-annotated dataset from the Reddit forum platform. This dataset contains rich context information including posts, comments, responses, and authors. Since our work focuses on text-based sarcasm recognition, we do not use this dataset.
Implementation Details. We use 300-dimensional Glove (Pennington et al., 2014) embeddings to initialize word vectors. There is a checkpoint every 16 mini-batch, and the batch size is 32. For Tweets dataset, the dropout on embeddings is set to 0, while for IAC datasets it is set to 0.5. Adam (Kingma and Ba, 2015) is used to optimize
<sup>3</sup>https://nlds.soe.ucsc.edu/sarcasm1
4https://nlds.soe.ucsc.edu/sarcasm2
<sup>5</sup>https://github.com/Cyvhee/SemEval2018-Task3
<sup>2</sup>http://www.4forums.com/political/
Table 2: The precision, recall, and macro  $F1$  of sarcasm recognition. The results marked with * are from Hee et al. (2018a). The best results are in boldface and second-best underlined.
Model	IAC-V1	IAC-V2	Tweets
Pre.	Rec.	F1	Acc.	Pre.	Rec.	F1	Acc.	Pre.	Rec.	F1	Acc.
UCDCC	58.6	58.6	58.5	58.5	67.1	67.0	67.0	67.0	78.8*	66.9*	72.4*	79.7*
THU-NGN	64.4	64.3	64.2	64.3	73.3	73.3	73.3	73.3	63.0*	80.1*	70.5*	73.5*
Bi-LSTM	64.6	64.6	64.6	64.6	79.8	79.7	79.7	79.7	71.8	71.7	71.7	73.0
AT-LSTM	65.9	65.5	65.3	65.5	76.7	76.2	76.1	76.2	70.8	71.6	70.0	70.2
CNN-LSTM-DNN	61.5	61.2	60.9	61.1	75.4	75.3	75.2	75.3	71.9	72.9	71.9	72.3
MIARN	65.6	65.2	64.9	65.2	75.4	75.3	75.2	75.3	68.6	68.8	68.8	70.2
ADGCN	64.3	64.3	64.3	64.3	81.0	80.9	80.9	80.9	72.6	73.2	72.8	73.6
DC-Net	66.6	66.5	66.4	66.5	82.2	82.1	82.1	82.1	76.4	77.5	76.3	76.7
our model. The parameters  $\beta_{1}$  and  $\beta_{2}$  of Adam are set to 0.9 and 0.999. The learning rates for model parameters except word vectors are 1e-3, and 1e-4 for word vectors. Our model is implemented with Pytorch $^{6}$  (version 1.7.0).
On IAC datasets, all of the loss contributions  $\lambda_{1},\lambda_{2},\lambda_{3}$  of our DC-Net model are set to 1. On Tweets, they are set to 1, 1e-4, and 3e-1, respectively. The hyperparameters are searched over the validation sub-dataset.
We evaluate our model against the following baselines:
UCDCC (Ghosh and Veale, 2018) is a siamese LSTM model exploiting Glove word embedding features. The method designs a lot of rules to preprocess Twitter data. It achieves the best performance on SemEval 2018 Task 3 Subtask A.
THU-NGN (Wu et al., 2018) consists of densely connected LSTMs based on word embeddings, sentiment features, and syntactic features. It ranks second on SemEval 2018 Task 3 Subtask A.
Bi-LSTM (Hochreiter and Schmidhuber, 1997) is a variant of RNN, which could learn long-term dependencies and bidirectional information.
AT-LSTM (Wang et al., 2016) is an LSTM model followed by a neural attention mechanism. It could attend the important part of the input.
CNN-LSTM-DNN (Ghosh and Veale, 2016) is a combination of CNN, LSTM, and DNN. It stacks two layers of convolution and two LSTM layers, then passes the output to a DNN for prediction.
MIARN (Tay et al., 2018) learns the intra-sentence
relationships of word pairs and the sequential relationships of a given text.
ADGCN (Lou et al., 2021) is a GCN-based method with sentic graph and dependency graph<sup>7</sup>. The initial input of GCN is the hidden state of Bi-LSTM.
4.3 Main Experiment Results
Table 2 shows that our DC-Net achieves the best macro  $F1$  results across all datasets. On Tweets dataset, DC-Net achieves about  $3.5\%$  improvement in  $F1$  score than the best baseline. On IAC-V2 dataset, our model outperforms the second-best by  $1.2\%$  in  $F1$ . Surprisingly, compared with the basic encoder model Bi-LSTM, our DC-Net boosts the performance up to  $5\%$  and  $3\%$  respectively on Tweets and IAC-V2, demonstrating the effectiveness of our dual-channel design. For Tweets dataset, the average length of texts is 14 words, which leads to a lack of information for sarcasm recognition. Nevertheless, our DC-Net improves  $3.5\%$  on  $F1$  compared with the previous state-of-the-art ADGCN.