SlowGuess/ACL_2020_OCR huggingface dataset:
Srijan Bansal<sup>1</sup>, G Vishal<sup>2</sup>, Ayush Suhane<sup>3</sup>, Jasabanta Patro<sup>4</sup>, Animesh Mukherjee<sup>5</sup>
Indian Institute of Technology, Kharagpur, West Bengal, India - 721302
$\{^1$ srijanbansal97, $^2$ vishal_g, $^3$ ayushsuhane99]@iitkgp.ac.in,
4jasabantapatro@iitkgp.ac.in, 5animeshm@cse.iitkgp.ac.in
In this paper we demonstrate how code-switching patterns can be utilised to improve various downstream NLP applications. In particular, we encode different switching features to improve humour, sarcasm and hate speech detection tasks. We believe that this simple linguistic observation can also be potentially helpful in improving other similar NLP applications.
Code-mixing/switching in social media has become commonplace. Over the past few years, the NLP research community has in fact started to vigorously investigate various properties of such code-switched posts to build downstream applications. The author in (Hidayat, 2012) demonstrated that inter-sentential switching is preferred more than intra-sentential switching by Facebook users. Further while  $45\%$  of the switching was done for real lexical needs,  $40\%$  was for discussing a particular topic and  $5\%$  for content classification. In another study (Dey and Fung, 2014) interviewed Hindi-English bilingual students and reported that  $67\%$  of the words were in Hindi and  $33\%$  in English. Recently, many down stream applications have been designed for code-mixed text. (Han et al., 2012) attempted to construct a normalisation dictionary offline using the distributional similarity of tokens plus their string edit distance. (Vyas et al., 2014) developed a POS tagging framework for Hindi-English data.
More nuanced applications like humour detection (Khandelwal et al., 2018), sarcasm detection (Swami et al., 2018) and hate speech detection (Bohra et al., 2018) have been targeted for code-switched data in the last two to three years.
1.1 Motivation
The primary motivation for the current work is derived from (Vizcañó, 2011) where the author notes – “The switch itself may be the object of humour”. In fact, (Siegel, 1995) has studied humour in the Fijian language and notes that when trying to be comical, or convey humour, speakers switch from Fijian to Hindi. Therefore, humour here is produced by the change of code rather than by the referential meaning or content of the message. The paper also talks about similar phenomena observed in Spanish-English cases.
In a study of English-Hindi code-switching and swearing patterns on social networks (Agarwal et al., 2017), the authors show that when people code-switch, there is a strong preference for swearing in the dominant language. These studies together lead us to hypothesize that the patterns of switching might be useful in building various NLP applications.
1.2 The present work
To corroborate our hypothesis, in this paper, we consider three downstream applications – (i) humour detection (Khandelwal et al., 2018), (ii) sarcasm detection (Swami et al., 2018) and (iii) hate speech detection (Bohra et al., 2018) for Hindi-English code-switched data. We first provide empirical evidence that the switching patterns between native (Hindi) and foreign (English) words distinguish the two classes of the post, i.e., humour vs non-humour or sarcastic vs non-sarcastic or hateful vs non-hateful. We then feature these patterns and pump them in the state-of-the-art classification models to show the benefits. We obtain a macro-F1 improvement of  $2.62\%$ ,  $1.85\%$  and  $3.36\%$  over the baselines on the tasks of humour detection, sarcasm detection and hate speech detection respectively. As a next step, we introduce a modern deep neu
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1018-1023 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
ral model (HAN - Hierarchical Attention Network (Yang et al., 2016)) to improve the performance of the models further. Finally, we concatenate the switching features in the last hidden layer of the HAN and pass it to the softmax layer for classification. This final architecture allows us to obtain a macro-F1 improvement of  $4.9\%$ ,  $4.7\%$  and  $17.7\%$  over the original baselines on the tasks of humour detection, sarcasm detection and hate speech detection respectively.
We consider three datasets consisting of Hindi (hi) - English (en) code-mixed tweets scraped from Twitter for our experiments - Humour, Sarcasm and Hate. We discuss the details of each of these datasets below.
+	-	Tweets	Tokens	Switching*
Humour	1755	1698	3453	9851	2.20
Sarcasm	504	4746	5250	14930	2.13
Hate	1661	2914	4575	10453	4.34
Table 1: Dataset description (* denotes average/tweet).
Humour: Humour dataset was released by (Khandelwal et al., 2018) and has Hindi-English code-mixed tweets from domains like 'sports', 'politics', 'entertainment' etc. The dataset has uniform distribution of tweets in each category to yield better supervised classification results (see Table 1) as described by (Du et al., 2014). Here the positive class refers to humorous tweets while the negative class corresponds to non-humorous tweet. Some representative examples from the data showing the point of switch corresponding to the start and the end of the humour component.
women can crib on things like humourstart bhaiyya ye shakkar bahot zyada meethi hai humourend, koi aur quality dikhao1
- shashi kapoor trending on mothersday how apt,  $\frac{\text{humour}_{\text{start}}}{2}$  mere paas ma hai  $\text{humour}_{\text{end}}$
political journey of kejriwal, from humourstart mujhe chahiye swaraj humourend to humourstart mujhe chahiye laluraj humourend3
Sarcasm: Sarcasm dataset released by (Swami et al., 2018) contains tweets that have hashtags #sarcasm and #irony. Authors used other keywords such as 'bollywood', 'cricket' and 'politics' to collect sarcastic tweets from these domains. In this case, the dataset is heavily unbalanced (see Table 1). Here the positive class refers to sarcastic tweets and the negative class means non-sarcastic tweets. Some representative examples from our data showing the point where the sarcasm starts and ends.
said aib filthy pandit ji,  $sarcasm_{start}$  aap jo bol rahe ho woh kya shuddh sanskrit hai  $sarcasm_{end}$ ? irony shameonyou4
- irony bippi lahiri sings  $\underline{\text{sarcasm}_{start}}$  sona nahi chandi nahi yaar toh mila arre pyaar kar le  $\underline{\text{sarcasm}_{end}}$
Hate speech: (Bohra et al., 2018) created the corpus using the tweets posted online in the last five years which have a good propensity to contain hate speech (see Table 1). Authors mined tweets by selecting certain hashtags and keywords from 'politics', 'public protests', 'riots' etc. The positive class refers to a hateful tweets while the negative class means non-hateful tweets<sup>6</sup>. An example of hate tweet showing the point of switch corresponding to the start and the end of the hate component.
- I hate my university,  $\underline{\text{hate}}_{\text{start}}$  koi us jagah ko aag laga dey  $\underline{\text{hate}}_{\text{end}}$ .
3 Switching features
In this section, we outline the key contribution of this work. In particular, we identify how patterns of switching correlate with the tweet text being humorous, sarcastic or hateful. We outline a synopsis of our investigation below.
3.1 Switching and NLP tasks
In this section, we identify how switching behavior is related to the three NLP tasks at our
Gloss: women can crib on things like brother the sugar is a little more sweet, show a different quality.
Gloss: shashi kapoor trending on mothersday how apt, I have my mother with me.
Gloss: political journey of kejriwal, from I want swaraj to I want laluraj.
Gloss: said aib filthy pandit ji, whatever you are telling is it pure sanskrit? irony shameonyou.
5irony bappi lahiri sings Gloss: doesn't matter you do not get gold or silver, you have got a friend to love.
The dataset released by this paper only had the hate/non-hate tags for each tweet. However, the language tag for each word required for our experiments was not available. Two of the authors independently language tagged the data and obtained an agreement of  $98.1\%$ . While language tagging, we noted that the dataset is a mixed bag including hate speech, offensive and abusive tweets which have already been shown to be different in earlier works (Waseem et al., 2017). However, this was the only Hindi-English code-mixed hate speech dataset available.
Gloss: I hate my university. Someone burn that place.
hand. Let  $\mathcal{Q}$  be the property that a sentence has en words which are surrounded by hi words, that is there exists an English word in a Hindi context. For instance, the tweet koi_hi to_hi pray_en karo_hi mere_hi liye_hi bhi_hi satisfies the property  $\mathcal{Q}$ . However, bumrah_hi dono_hi wicketo_hi ke_hi beech_hi gumrah_hi ho_hi gaya_hi does not satisfy  $\mathcal{Q}$ .
We performed a statistical analysis to determine the correlation between the switching patterns and a classification task at hand (represented by  $\mathcal{T}$ ). Let us denote the probability that a tweet belongs to a positive class for a task  $\mathcal{T}$  given that it satisfies property  $\mathcal{Q}$  by  $p(\mathcal{T}|\mathcal{Q})$ . Similarly, let  $p(\mathcal{T}|\sim \mathcal{Q})$  be the probability that the tweet belongs to the positive class for task  $\mathcal{T}$  and does not satisfy the property  $\mathcal{Q}$ .
Further let  $avg(S|\mathcal{T})$  be the average switching in positive samples for the task  $\mathcal{T}$  and  $avg(S|\sim \mathcal{T})$  denote the average switching in negative samples for the task  $\mathcal{T}$ .
T: Humour	T: Sarcasm	T: Hate
p(T|Q)	0.56	0.28	0.36
p(T|~Q)	0.50	0.42	0.41
avg(S|T)	7.84	0.60	1.49
avg(S|~T)	6.50	0.89	1.54
The main observations from this analysis for the three tasks – humour, sarcasm and hate are noted in Table 2. For the humour task,  $p(\text{humour} \mid \mathcal{Q})$  dominates over  $p(\text{humour} \mid \sim \mathcal{Q})$ . Further the average number of switching for the positive samples in the humour task is larger than the average number of switching for the negative samples. Finally, we observe a positive Pearson's correlation coefficient of 0.04 between a text being humorous and the text having the property  $\mathcal{Q}$ . This together indicates that the switching behavior has a positive connection with a tweet being humorous.
On the other hand  $p(\text{sarcasm} \mid \sim \mathcal{Q})$  as well as  $p(hate \mid \sim \mathcal{Q})$  respectively dominate over  $p(\text{sarcasm} \mid \mathcal{Q})$  and  $p(hate \mid \mathcal{Q})$ . Moreover the average number of switching for the negative samples for both these tasks is larger than the average number of switching for the positive samples. The Pearson's correlation between a text being sarcastic (hateful) and the text having the property  $\mathcal{Q}$  is negative: -0.17 (-0.04). This shows there is an overall negative connection between the switching behavior and sarcasm/hate speech detection tasks.
Table 2: Correlation of switching with different classification tasks.
Feature name	Description
en_hi SWITCHES	The number of en to hi switches in a sentence
hi_en SWITCHES	The number of hi to en switches in a sentence
V	The total number of switches in a sentence
fraction_en	Fraction of English words in a sentence
fraction_hi	Fraction of Hindi words in a sentence
mean_hi_en	Mean of hi_en vector
stddev_hi_en	Standard deviation of hi_en vector
mean_en_hi	Mean of en_hi vector
stddev_en_hi	Standard deviation of en_hi vector
Table 3: Description of the switching features.
While we have tested on one language pair (Hindi-English), our hypothesis is generic and has been already noted by linguists earlier (Vizcaño, 2011).
3.2 Construction of the feature vector
Motivated by the observations in the previous section we construct a vector  $\mathbf{hi\_en}[i]$  that denotes the number of Hindi (hi) words before the  $i^{\text{th}}$  English (en) word and a vector  $\mathbf{en\_hi}[i]$  that denotes the number of English (en) words before the  $i^{\text{th}}$  Hindi (hi) word. This can also be interpreted as the run-lengths of the Hindi and the English words in the code-mixed tweets. Based on these vectors we define nine different features that capture the switching patterns in the code-mixed tweets<sup>8</sup>.
An example feature vector computation: Consider the sentence - koi_hi to_hi pray_en karo_hi mere_hi liye_hi bhi_hi.
$$
\begin{array}{l} \text {h i} _ {\text {e n}} \\ : [ 0, 0, 2, 0, 0, 0, 0 ] \end{array}
$$
$$
\begin{array}{l} \text {e n} _ {\mathbf {h}} \mathbf {i} \\ : [ 0, 0, 0, 1, 1, 1, 1 ] \end{array}
$$
$$
\text {F e a t u r e v e c t o r :} [ 1, 1, 2, \frac {1}{7}, \frac {6}{7}, \frac {2}{7}, 0. 6 9, \frac {4}{7}, 0. 4 9 ]
$$
4.1 Pre-processing
Tweets are tokenized and punctuation marks are removed. All the hashtags, mentions andurls are stored and converted to string 'hashtag', 'mention' and 'url' to capture the general semantics of the tweet. Camel-case hashtags were segregated and included in the tokenized tweets (see (Belainine et al., 2016), (Khandelwal et al., 2017)). For example, #AadabArzHai can be decomposed into three distinct words: Aadab, Arz and Hai. We use the same pre-processing for all the results presented in this paper.
<sup>8</sup>We tried with different other variants but empirically observe that these nine features already subsumes all the necessary distinguishing qualities.
4.2 Machine learning baselines
Humour baseline (Khandelwal et al., 2018): Uses features such as  $n$ -grams, bag-of-words, common words and hashtags to train the standard machine learning models such as SVM and Random-Forest. The authors used character  $n$ -grams, as previous work shows that this feature is very efficient in classifying text because they do not require expensive text pre-processing techniques like tokenization, stemming and stop words removal. They are also language independent and can be used in code-mixed texts. In their paper, the authors report the results for tri-grams.
Sarcasm baseline (Swami et al., 2018): This model also uses a combination of word  $n$ -grams, character  $n$ -grams, presence or absence of certain emoticons and sarcasm indicative tokens as features. A sarcasm indicative score is computed and chi-squared feature reduction is used to take the top 500 most relevant words. These were incorporated into features used for classification. Standard off-the-shelf machine learning models like SVM and Random Forest were used.
Hate baseline (Bohra et al., 2018): The hate speech detection baseline also consists of similar features such as character  $n$ -grams, word  $n$ -grams, negation words and a lexicon of hate indicative tokens. Chi-squared feature reduction method was used to decrease the dimensionality of the features. Once again SVM and Random Forest based classifiers were used for this task.
Switching features: We plug in the nine switching features introduced in the previous section to the three baseline models for humour, sarcasm and hate speech detection.
4.3 Deep learning architecture
In order to draw the benefits of the modern deep learning machinery, we build an end-to-end model for the three tasks at hand. We use the Hierarchical Attention Network (HAN) (Yang et al., 2016) which is one of the state-of-the-art models for text and document classification. It can represent sentences in different levels of granularity by stacking recurrent neural networks on character, word and sentence level by attending over the words which are informative. We use the GRU implementation of HAN to encode the text representation for all
the three tasks.
Handling data imbalance by sub-sampling: Since the sarcasm dataset is heavily unbalanced we sub-sampled the data to balance the classes. To this purpose, we categorise the negative samples into those that are easy or hard to classify. Hypothesizing that if a model can predict the hard samples reliably it can do the same with the easy samples. We trained a classifier model on the training dataset and obtained the softmax score which represents  $p(\text{sarcastic|text})$  for the test samples. Those test samples which have a score less than a very low confidence score (say 0.001) are removed imagining them to be easy samples. The dataset thus got reduced and more balanced. It is important to note that positive samples are never removed. We validated this hypothesis through the test set. Our trained HAN model achieves an accuracy of  $94.4\%$  in classifying the easy (thrown out) samples as non-sarcastic thus justifying the sub-sampling.
Switching features: We include the switching features to the pre-final fully-connected layer of HAN to observe if this harnesses additional benefits (see Figure 1).
Figure 1: The overall HAN architecture along with the switching features in the final layer.
Train-test split: For all datasets, we maintain a train-test split of 0.8 - 0.2 and perform 10-fold cross validation.
Parameters of the HAN: BiLSTMs: no dropout, early stopping patience: 15, optimizer = 'adam' (learning rate = 0.001, beta_1 = 0.9), loss = binary cross entropy, epochs = 200, batch_size = 32, pretrained word-embedding size = 50, hidden size: [20, 60], dense output size (before concatenation):
9see Christopher Pott's sentiment tutorial: http://sentiment.christopherpotts.net/ lingstruc.html
Model	Humour	Sarcasm	Hate
Baseline (B)	69.34	78.4	33.60
Baseline + Feature (BF)	71.16	79.85	34.73
HAN (H)	72.04	81.36	38.78
HAN + Feature (HF)	72.71	82.07	39.54
Table 4: Summary of the results from different models in terms of macro-F1 scores. M-W U test shows all improvements of HF over B are significant.
[15,30].
Pre-trained embeddings: We obtained pretrained embeddings by training GloVe from scratch using the large code-mixed dataset (725173 tweets) released by (Patro et al., 2017) plus all the tweets (13278) in our three datasets.
We compare the baseline models along with (i) the baseline + switching feature-based models and (ii) the HAN models. We use macro-F1 score for comparison all through. The main results are summarized in Table 4. The interesting observations that one can make from these results are - (i) inclusion of the switching features always improves the overall performance of any model (machine learning or deep learning) for all the three tasks, (ii) the deep learning models are always better than the machine learning models. Inclusion of switching features into the machine learning models (indicated as BF in Table 4) allows us to obtain a macro-F1 improvement of  $2.62\%$ ,  $1.85\%$  and  $3.36\%$  over the baselines (indicated as B in Table 4) on the tasks of humour detection, sarcasm detection and hate speech detection respectively. Inclusion of the switching feature in the HAN model (indicated as HF in Table 4) allows us to obtain a macro-F1 improvement of  $4.9\%$ ,  $4.7\%$  and  $17.7\%$  over the original baselines (indicated as B in Table 4) on the tasks of humour detection, sarcasm detection and hate speech detection respectively.
Success of our model: Success of our approach is evident from the following examples. For instance, as we had demonstrated earlier, humour is positively correlated with switching, a tweet having a switching pattern like - anurag_hi kashyap_hi can_en never_en join_en aap_hi because_en ministers_en took_en oath_en, "main_hi kisi_hi anurag_hi aur_hi dwesh_hi ke_hi bina_hi kaam_hi karunga_hi" which was not detected as humorous by the baseline (B) but was detected so by our models (BF and HF). Note that the author of the above tweet seems to have categorically switched
to Hindi to express the humour; such observations have also been made in (Rudra et al., 2016) where opinion expression was cited as a reason for switching.
Sarcasm being negatively correlated with switching, a tweet without having switching is more likely to be sarcastic. For instance, the tweet naadaan_hi baalak_hi kalyug_hi ka_hi vardaan_hi hai_hi ye_hi, which bears no switching was labeled non-sarcastic by the baseline. Our models (BF and HF) have rectified it and correctly detected it as sarcastic.
Similarly, hate being negatively correlated with switching, a tweet with no switching - shilpa_hi ji_hiAAP_hi ravidubey_hi jaise_hi tuchho_hi ko_hi jawab_hi mat_hi dijiye_hi ye_hi log_hi aap_hi ke_hi sath_hi kabhi_hi nahi_hi was labeled as non-hateful by the baseline, was detected as hateful by our methods (BF and HF).
In this paper, we identified how switching patterns can be effective in improving three different NLP applications. We present a set of nine features that improve upon the state-of-the-art baselines. In addition, we exploit the modern deep learning machinery to improve the performance further. Finally, this model can be improved further by pumping the switching features in the final layer of the deep network.
In future, we would like to extend this work for other language pairs. For instance, we have seen examples of such switching in English-Spanish $^{10}$  and English-Telugu $^{11}$  pairs also. Further we plan to investigate other NLP applications that can benefit from the simple linguistic features introduced here.
P. Agarwal, A. Sharma, J. Grover, M. Sikka, K. Rudra, and M. Choudhury. 2017. I may talk in english but gaali toh Hindi mein hi denge: A study of english-hindi code-switching and swearing pattern on social networks. In 2017 9th International Conference on Communication Systems and Networks (COM-SNETS), pages 554-557.
<sup>10</sup> Don't forget humour start la toalla cuando go to le playa humour end; Gloss: Don't forget the towel when you go to the beach.
<sup>11</sup>Votes kosam quality leni liquor bottles supply chesina politicians sarcasmstart ee roju quality gurinchi matladuthunnaru sarcasmend; Gloss:Politicians who supplied low quality liquor bottles for votes are talking about quality today.