Sarcasm detection research of the Bengali language so far can be considered to be narrow due to the unavailability of resources. In this paper, we introduce a large-scale self annotated Bengali corpus for sarcasm detection research problem in the Bengali language named ’Ben-Sarc’ containing 25,636 comments, manu- ally collected from different public Facebook pages and evaluated by external evaluators. Then we present a complete strategy to utilize different models of traditional machine learning, deep learning, and transfer learning to detect sarcasm from text using the Ben-Sarc corpus. Finally, we demonstrate a com- parison between the performance of traditional machine learning, deep learning, and transfer learning models on our Ben-Sarc corpus. Transfer learning using Indic-Transformers Bengali BERT as a pre-trained source model has achieved the highest accuracy of 75.05%. The second highest accuracy is obtained by the LSTM model with 72.48% and Multinomial Naive Bayes is acquired the third highest with 72.36% accuracy for deep learning and machine learning, respec- tively. The Ben-Sarc corpus is made publicly available in the hope of advancing the Bengali Natural Language Processing community. Corresponding author Email addresses: sanzanalora@yahoo.com (Sanzana Karim Lora), shahariar_shibli.cse@aust.edu (G. M. Shahariar), tamanna.naz98@gmail.com (Tamanna Nazmin), nafirahman27@gmail.com (Noor Nafeur Rahman), rafsanrahman35549@gmail.com (Rafsan Rahman ), miyadbhuiyan@gmail.com (Miyad Bhuiyan), faisal.cse@aust.edu (Faisal Muhammad Shah) Preprint submitted to Computer Speech and Language January 4, 2022 Keywords: Sarcasm, Bengali sarcasm, sarcasm detection, Bengali sarcasm detection 1. INTRODUCTION An ironic, stinging, sour, cutting statement or comment that indicates the reverse of what someone truly intends to express is Sarcasm [1]. The use of sarcastic language is a resentment concealed as humor and intended to provoke, annoy, or convey contempt. As the intention of sarcasm is often vague and mis-5 leading, people cannot discriminate between a true story and satire or irony [1]. Facebook, YouTube, and Twitter are influential social media platforms for shar- ing people’s judgments, thoughts, opinions, sentiments nowadays [2]. The afore- mentionedlargeamountofavailabledataofferstheextenttoresearchinNatural Language Processing (NLP).10 Sarcasm detection in low resource language is a very narrow research area in Natural Language Processing. Sarcasm detection is a subset of sentiment anal- ysis problems where the focus is on recognizing sarcasm rather than identifying a sentiment across the board [3]. Sarcasm detection researches are available for15 high resource languages such as English. But, despite being the world’s seventh most spoken language with 240 million native speakers [4], research on sarcasm detection in the Bengali language is unexplored and overlooked. Due to the lim- ited resources and the scarcity of large-scale sarcasm data, identifying sarcasm from Bengali text is currently a difficult challenge for the researchers of NLP [5].20 Facebook is a popular free social networking website that allows registered users to upload photos, videos, send messages and keep in touch with friends, family, and colleagues1. Bangladesh has 41 million Facebook users since Jan- uary 20212. People socialize in the Facebook comment section to express their 25 1https://whatis.techtarget.com/definition/Facebook/ 2https://www.statista.com/statistics/268136/top-15-countries-based-on-number-of-facebook-users/ 2 perspectives, judgments, and opinions on the content of a post. Any automatic detection system that uses machine learning is large-scale dataset dependent as it requires rigorous training and testing. As far as we have noticed, there is no available Bengali text corpus for sarcasm detection. We have constructed a corpus named ’Ben-Sarc’ that contains Facebook comments written in Bengali.30 Furthermore, we have classified the Bengali texts as Sarcastic and Non-sarcastic and proposed a sarcasm detection model using machine learning. Our main contributions in this paper are summarized as follows: •At first, we have constructed a large-scale self annotated Bengali corpus for sarcasm detection. The corpus can be found at https://shorturl. 35 at/oFJRZ . •Next, we have evaluated our constructed corpus by external human eval- uators who are experts in this field. •Then, we have conducted a comprehensive experiment on this corpus to detect sarcasm from Bengali texts with the help of traditional Machine 40 Learning, Deep Learning, and Transfer Learning approaches to set a base- line for future researchers. In the next section, we briefly discuss related works on high and low resource language sarcasm detection. Section 3 shows the dataset creation along with theannotationprocess. Moreover, Section4explainstheproposedmethodology.45 Section 5 contains the experimental results and their analysis, while Section 6 contains the conclusion and future work. 2. RELATED WORKS The increasing engagement of social media users influences the quantitative and qualitative analysis of available data. Though most of the research is on50 the English language, sarcasm detection for low resource languages such as In- donesian [6], Hindi [7] [8] [9], Czech [10], and Japanese [11] are available. We 3 discusssomeoftherelatedapproachesinthefollowingliteraturereviewanalysis. 2.1. English Language55 [9]experimentedwithtraditional machinelearning algorithmssuchasSVM, KNN, and Random forest on 9104 tweets on Twitter. [12] worked on both sar- casm and irony detection separately. SVM, Naive Bayes (NB), Decision Tree, Random Forest (RF) were applied on the irony detection dataset whereas SVM and Random Forest (RF) algorithms were on the sarcasm detection dataset.60 The segregated experiments gained 64% accuracy on irony and 76% accuracy on sarcasm detection. There exist a few models that use contextual information regarding the tweets on Twitter to detect sarcasm. [13] focused on the context of authors65 and audiences on Twitter posts to figure out sarcastic content with 85.1% ac- curacy. Binary logistic regression was applied to train the model upon 19534 tweets. [14] also aimed at the context for identifying sarcasm accurately. They collected 1500 tweets and derived 6774 history-based, 453 conversation-based, 2618 topic-based contextual tweets. Sequential SVM classifier exhibited a de-70 cent accuracy of 69.13%. [15] extracted 5000 tweets that include texts, labels, and contexts and analyzed the dataset through linear SVC, Logistic Regression (LR), Gaussian Naive Bayes (GNB), and Random Forest (RF) classifiers. They utilized BERT and GloVe embeddings to the algorithms. Logistic Regression with GloVe embeddings gained 69% accuracy on the dataset that involves con-75 text. Hashtags exhibit a meaningful role in the content on Twitter. [9] extracted 9104 tweets containing hashtags such as “#sarcasm” and “#not” in Hindi and English. They implemented three SVM, KNN, and Random Forest (RF) clas-80 sifiers. Random Forest (RF) showed an 81% accuracy on sarcasm detection. 4 [1] considered the impact of positive and negative situations on different sentiments to analyze sarcasm. They used a supervised SVM classifier and an N-gram classifier. To increase the accuracy, they optimized the RBF kernel,85 cost, and gamma parameters over 35000 tweets. [16] inflicted four models: bidirectional LSTM, LSTM and CNN, SVM, and Multi-layer perception on 9400 data collected from Reddit and Twitter. Each model used 10-fold cross-validation. The ensemble method achieved the best90 F1 score. Very few research works executed deep learning models alongside the transformers models to improve the accuracy of the prediction of sarcasm de- tection models. 2.2. Bengali Language95 Recently, emotion and specific sentiment analysis tasks like abusive text detection, toxicity detection, hateful speech detection from Bengali text have received extra attraction to many researchers involved in the Bengali Language Processing area. 100 [17] presented a deep learning approach to detect sentiment labels and emotions from Bengali, Romanized Bengali, and English YouTube comments. Skip-Gram and a continuous bag of words (CBOW) in Word2Vec are used to get the word embedding representation for CNN and LSTM model. 105 [18] presented a machine learning-based model and GRU-based deep neural network model to detect hateful speech from Facebook public pages’ comments where GRU obtained a 70.10% accuracy. They collected 5126 comments, anno- tated them, and divided them into six classes. 110 [19] reported a deep learning approach for detecting abusive Bengali com- ments. Using RNN on 4700 Bengali text documents, they achieved an accuracy 5 of 82%. [2] used 300 Facebook comments without using any predictive al- gorithm to detect abusive Bengali text. [20] used Multinomial Naive Bayes (MNB), SVM, and Linear SVM to identify offensive text from 5644 posts and115 comments with emoticons where Linear SVM achieved 78% accuracy. [21] col- lected 2665 English texts from Youtube and translated them into Bengali to build the abusive text dataset. Naive Bayes (NB) classifier achieved 80.57% accuracy with a 39% f1 score using a 10 fold cross-validation. 120 [8] identified aggression and misogynistic aggression from English, Hindi, and Bengali text. They utilized En-BERT, RoBERTa, DistilRoBERTa, SVM for the English language but M-BERT, XLM RoBERTa, SVM for Bengali, and Hindi. [22] detected cyberbullying from Bengali text by NB, KNN, SVM us- ing 2400 Bengali text collected from Facebook and Twitter. [23] tried machine125 learninganddeeplearningmodelsfortoxicitydetectionusing4255Bengalicom- ments. Thelimitationsofalltheseworkssymbolizetheunavailabilityofalarge-scale Bengalitextcorpus. Forthisreason, [24]constructedadatasetcontaining44001130 Facebook public posts’ comments for helping the researchers to detect online harassment. There is a limited number of contributions in the area of Satire, irony, or sar- casm detection. [25] detected satire in Bengali documents. They created their135 own Word2Vec model and achieved an accuracy of 96.4% by using the CNN model but the dataset had insufficient data. [26] identified sarcasm from 41350 Facebook posts considering public reactions and interactive comments and im- ages. They utilized machine learning algorithms and a CNN-based model to detect sarcasm from images. Though the dataset is adequately large, the anno-140 tation process should have received special attention. As far as we have seen, there is no comprehensive study that utilizes machine 6 Table 1: The Content of the Facebook Pages News Channels News Papers TV Channels Public Figures Miscellaneous Somoynews.tv Prothom Alo Zee Bangla Shakib Al Hasan Bdcrictime.com Jamuna Television Bangla Tribune Star Jalsha Pori Moni Amari Dhaka Ekattor Udvash BBC News Bangla Lords Association learning, deep learning and transfer learning to detect sarcasm. Therefore, in this paper, we have presented a comprehensive approach that includes machine145 learning, deep learning, and transfer learning. Besides, we have introduced a large-scale human-annotated dataset named ”Ben-Sarc” containing 25636 com- ments written in Bengali collected from Facebook. 3. DATASET CONSTRUCTION150 Asfaraswehaveseen, thereisnoavailablelabeleddatasetforsarcasmdetec- tion in Bengali. We felt the need to create our sarcasm detection dataset for the Bengali language. We defined our dataset as the Bengali Sarcasm dataset (Ben- Sarc). The duration of dataset construction is approximately three months. In the following subsections, we discuss the features of our Ben-Sarc dataset in155 detail. 3.1. Content Source AsFacebookisoneofthemajorsourcesoftextualdata[27], wehavetargeted public Facebook pages to construct the Ben-Sarc dataset. We have collected160 Bengali Facebook comments from 14 different public pages from Bangladesh and India dated from 2013 to 2021. The content of the pages is shown in the table 1. 3.2. Content Search Facebook comment section usually consists of the reaction of users based on 165 the post. The commenters of targeted pages are mostly Bengali language people 7 and there are lots of comments written in Bengali, English, and Romanized Bengali. We have only taken the Bengali comments. All the comments have been scrapped manually by the authors of this paper. 3.3. Text Cleaning and Noise Removal170 Text preprocessing is generally a vital phase of natural language processing (NLP) problems [28]. It converts text into a convenient format. The comment section of Facebook is very noisy and mostly contains errors, useless informa- tion [27]. A list of pre-processing steps has been executed on the texts col- lected to enrich the Ben-Sarc dataset. They are - removing non-Bengali words,175 duplicate texts, emojis, links, URLs; replacing #hashtag, all symbols, special characters (e.g. ‘\n’, ‘%’, ‘$’, ‘&’, ‘@’) with a single space and multiple punctu- ations(e.g. ‘?’, ‘ ।’, ‘;’, ‘!’, ‘,’) with single punctuation. 3.4. Annotation Process Each text in the Ben-Sarc dataset has been annotated manually by us using 180 ‘0’ and ‘1’ as we intend to work on a binary classification problem - sarcasm detection. ‘0’ means non-sarcastic comments and ‘1’ represents sarcastic com- ments. EachtextintheBen-Sarcdatasethasbeenannotatedbyfiveannotators. The final choice on the polarity of a single text has been made using the ma- jority voting method from five annotations. Facebook comments are frequently185 filled with harsh and filthy phrases, slang, and personal attacks [2] [24] [22]. As a result, we made sure that all annotators are of adult age and have domain knowledge. 3.5. Human Evaluation of Ben-Sarc Dataset To maintain the quality of a labeled dataset, evaluation is a necessary step. 190 We have tried to make sure the data in the Ben-Sarc dataset is not labeled vaguely keeping in mind that the researchers can use it for further applications without hesitation. The assessment process has been carefully accomplished in the Ben-Sarc dataset by two external human evaluators experts in this field. 8 Table 2: Inter Annotator Agreement of Ben-Sarc Assessed by Human Evaluators Questions Q1 in (%) Q2 in (%) Q3 in (%) Q4 in (%) Cohen’s Kappa Score 98.16 87.65 16.32 30.88 Each evaluator is an adult, native Bengali speaker, and proficient in Bengali.195 Each evaluator has been provided the task of assessing the quality of the dataset by replying ‘Yes’ or ‘No’ to the given questions stated below: Q1.Is the text ironic, caustic, or biting without emoji and emoticons? Q2.If Q1 is ‘Yes’, is the text written in the dialect, contains spelling mistakes or manipulated traditional phrases, sentences, songs, poems?200 Q3.If Q1 is ‘Yes’, is there any totally opposite context in that text? Q4.If Q1 is ‘Yes’, is there any information in the text that causes confusion to decide whether the text is sarcastic or not? The motivation of designing the questions for human evaluation of the Ben- 205 Sarc dataset is from [29]. The recent advancement in the quality estimation of neural language generation (NLG) models has inspired the creation of these characteristics. [30] demonstrated that NLG models are sensitive to low-quality training samples. Thus, it is critical to evaluate the quality of comments us- ing the characteristics of Q1. Moreover, to verify actual uniformity and fidelity,210 characteristics of Q2 and Q3 have been designed whereas Q4 determines if there is any ambiguous text or confusion to decide the polarity of the text. The text “আপনা�দর��কানা টা একট ু�দন, গা�ল�ল�খএকটা�����পাস্ট করব(Please give me your address, I will post a letter with obscenities) ” creates confusion because someone may take it as an abusive text, or a threat which leads it to a non-sarcastic text215 where others may take it as a joke that leads to sarcasm. Theinter-annotatoragreementismeasuredusingCohen’skappacoefficient[31] in table 2. Cohen’s kappa measures annotator agreement and determines how well one annotator agrees with another. To evaluate the conventional inter-220 9 annotator agreement, a pairwise kappa coefficient is computed using equation 1. K=PoPe 1 +Pe(1) where Porepresents relative observed agreement and Pedenotes the hypothe- sized probability of chance agreement. The quality assessment of Ben-Sarc is done on 5000 random samples of Ben-Sarc data. In most cases, the evaluators225 agree that the text seems ironic without any emoticons. Besides, a high percent- age for Q2 indicates that dialect, manipulation of the traditional poems, songs, and spelling mistakes also express sarcasm from the text whereas a low percent- age for Q3 determines the opposite context that is pretty normal. However, the Q4 raises an ambiguity to decide whether the text is sarcastic or not. In our230 situation, Q3 and Q4 should be in a very low percentage but the percentage of Q4 is comparatively higher than Q3 according to the inter-annotator agreement. 3.6. Dataset Description AdetaileddescriptionofourBen-Sarcdatasethasbeenpresentedinthissec- tion. The dataset contains a total of 25,636 Bengali comments where 12818 are235 sarcastic and 12818 are non-sarcastic. The visualization of the data distribution according to the labels is shown in fig 1. Figure 1: Data Distribution of Ben-Sarc Dataset According to Labels Table 3 represents a short overview of our labeled dataset construction. The maximum length of a text in the Ben-Sarc dataset is 395 in words and the 10 Table 3: A Overview of the Ben-Sarc Dataset Preprocessing Raw Text Preprocessed Text Polarity আমার মন বল �ছবাংলা�দশ�জত�ব। আমার মন বল �ছবাংলা�দশ�জত�ব। 0 বল�ছ�কএত ভালবাসা রাখ �বা�কাথায়??? দয়া ক �রবা���তভালবাসা রাখবার জ �ন��কছুপাত্র পা���য় �দ�বন…বল�ছ�কএত ভালবাসা রাখ �বা�কাথায়? দয়া ক �রবা���তভালবাসা রাখবার জ �ন��কছুপাত্র পা���য়�দ�বন ।1 এও একটা না �য়কা আর কাকও একটা গা �য়কা hahaha এও একটা না �য়কা আর কাকও একটা গা �য়কা 1 ক�রানা ভাইরাস ওহ না �কআ�বদন কর�ছ?? :p ক �রানা ভাইরাস ওহ না �কআ�বদন কর�ছ? 1 খুব ভাল অন ু��রণার এক #সা �কব! খ ুব ভাল অন ু��রণার এক সা �কব! 0 এই নয়া দামান গা�নর স���ন���ছ�লন ঢাকা�ম�ড�ক�লর ডাক্তাররাও, �কন তারা ওই �ভ�ডও বা�ন�য়�ছ�লন জান�তপার�বন এই�ভ�ডও�ত-->https://www.youtube.com/watch?v=এই নয়া দামান গা�নর স���ন���ছ�লন ঢাকা�ম�ড�ক�লর ডাক্তাররাও, �কন তারা ওই �ভ�ডও বা�ন�য়�ছ�লন জান�তপার�বন এই�ভ�ডও�ত।0 geIKbjOBZtM �বসম্ভব সুন্দর ছ�ব���� �বসম্ভব সুন্দর ছ�ব 1 Table 4: Overall Summary of Ben-Sarc Dataset Number of Comments Number of Words Number of Unique Words Sarcasm 12818 195445 28056 Non-Sarcasm 12818 184535 24838 Total 25636 379980 52894 minimum is 3 in words. Thus, the average length of a comment is 15. The240 length-frequency distribution of the whole dataset has been shown in fig 2. For better visualization, the length of the text has limited to 100. The overall summary of the Ben-Sarc dataset including the number of comments, words, and unique words according to its classes has been shown in table 4. The visualization of the statistics of the Ben-Sarc dataset has been shown in fig 3. Figure 2: Length-Frequency Distribution of the Ben-Sarc Dataset 245 11 Figure 3: The Visualization of the Statistics of the Ben-Sarc Dataset 4. PROPOSED METHODOLOGY In this section, we present our proposed methodology for sarcasm detection in brief. Figure 4 represents our proposed approach. We have distributed our proposed approach into five phases. The first phase comprises dataset construc- tion. The second phase involves dataset preprocessing by utilizing a few natural250 language preprocessing techniques like punctuation removal and tokenization. Thethirdphaseincorporatesthefeatureselectionprocess. Thisprocessincludes TF-IDF (Term-Frequency, Inverse Document Frequency) and n-grams for tradi- tional machine learning models, word embeddings for deep learning models, and pre-trained transformer-based models for transfer learning. The fourth phase of255 our proposed method is the training phase. In this phase, we have employed tra- ditional machine learning models, deep learning models, and transfer learning to classify text as sarcastic or non-sarcastic. We have examined the performance of each classifier and presented the best-performed classifier in the last phase. The details of all the phases are discussed in the following subsections.260 4.1. Phase I - Dataset Construction We have collected 25636 Facebook comments written in Bengali. The overall dataset construction process is described in section 3. 12 Figure 4: Work Flow of the Proposed Approach of This Paper 4.2. Phase II - Preprocessing A few pre-processing steps have been executed before model training which 265 is - punctuation removal (e.g. ‘!’, ‘?’), tokenization. Elongated words often contain some sentiment information. For example, 13 “খুউউব মজাআআআর (Veryyyyfunnnnny) ”emphasizesmorepositivesentimentthan “খুব মজার(Very funny)” [17]. So, we have not applied stemming and lemmati- zation to preserve the actual sense of the elongated words.270 4.3. Phase III - Feature Selection Feature selection is the third phase of our proposed model. We have used three feature extraction approaches: n-grams, TF-IDF, and word embeddings. For traditional machine learning classifiers, we have used TF-IDF and n-grams methods. TF-IDF is the most extensively utilized traditional feature extractor275 approach in classification applications [32]. It is a mathematical statistic that reveals to us how essential a term is to a document in a collection. The increase in a word’s TF-IDF value is directly proportional to the number of times that term appears in the document but is offset by the frequency of the term in the corpus, which helps to balance out terms that come more commonly in general.280 (TF−IDF )t;d=tft;d∗logN d ft(2) where, tft;dindicatesfrequencyofterm tindocument d,d ftdefinestotalnumber of documents containing term t,Nmeans the number of documents. To pick the features for deep learning models different pre-trained word embedding for Bengaliisused. Allareexplainedindetailinsection4.4.2. Theselectedfeatures oftransferlearningarepre-trainedlanguagemodelsdescribedindetailinsection285 4.4.3. 4.4. Phase IV - Training To classify whether a text is sarcastic or not, we have investigated traditional classifiers, deep learning classifiers, and transfer learning techniques. A compre- hensive description of all the models is manifested in the following subsections.290 4.4.1.Traditional Classifiers We have initiated the sarcasm detection system by investigating traditional classifiers. WehaveusedLogisticRegression(LR),DecisionTree(DT),Random 14 Forest (RF), Multinomial Naive Bayes (MNB), K Nearest Neighbors (KNN), Linear Support Vector Machine (SVM), and Kernel SVM as traditional machine295 learning classifiers. Furthermore, we have applied all possible combinations of unigram, bigrams, and trigrams by extracting the features using TF-IDF for both 5 and 10 fold cross-validation. The traditional classifiers are incapable of capturing the sequential informa- 300 tion present in the text. Besides, these are unsuitable for enhancing perfor- mance with a large number of data. So, we will experiment the performance of the Ben-Sarc dataset with deep learning models in the later. 4.4.2.Deep Learning Models Recurrent neural networks (RNNs) are deep learning neural networks that 305 are specially built to learn data sequences and are mostly used for textual data categorization. The learning process is carried out at hidden recurrent nodes based on their prior layers of nodes. However, when dealing with long sequences of data, RNNs suffer from the vanishing gradient problem. Long Short Term Memory (LSTM) [33] networks are a form of a recurrent neural network capable310 of learning order dependency in sequence prediction applications. LSTM has introduced a solution to the vanishing gradient problem and has shown to be efficient in various NLP-related applications. So, LSTM is chosen as our base- line model. Then, the LSTM model is expanded to understand the network’s behavior.315 4.4.2.1 Required Basic Components for Sarcasm Detection Models .In this subsection, the basic components of sarcasm detection models are ex- plained. If the reader is knowledgeable about these components, this subsection can be omitted. •LSTM: LSTMs interpret input sequences as pairs (xi; yi)::::(xz; yz). An 320 LSTM maintains a hidden vector htand a memory vector mtfor each pair (xi; yi)and at each time step t, which are responsible for regulating 15 state updates and outputs to create a target output yidepending on the previous state of the xiinput. At time step t, the computations are as follows [34] [35]: 325 ht=f(Wxt+Uht−1+b) (3) it=(Wixt+Uiht−1+bi) (4) ft=(Wfxt+Ufht−1+bf) (5) ot=(Woxt+Uoht−1+bo) (6) gt=(Wgxt+Ught−1+bg) (7) 330 ct=ft ct1 +it gt (8) ht=ot tanh (ct) (9) where indicates sigmoid function and indicates element-wise multipli- cation. Wi,Ui, and biare two weight matrices and a bias vector for input gate irespectively. The meaning is the same as for forget gate f, output gate o, tanh layer u, memory cell c, and hidden state h. The forget gate 335 selects which past information should be forgotten on its own, whereas the input gate decides what new information should be placed in the memory cell. Finally, the output gate determines how much information from the internal memory cell is revealed. This gate unit assists an LSTM model in remembering important information over numerous time steps. 340 •CNN:A CNN [36] is mainly made up of convolutional layers and pooling layers. The convolutional layers include weights that must be taught, whereas the pooling layers change the activation using a fixed function. –Convolutional Layer: A convolutional layer is made up of a num- ber of kernels whose parameters must be learnt. It is a local feature 345 extractor layer with well-trained kernels for weight modification uti- lizing the back-propagation approach [37]. The kernels’ height and 16 weight are less than those of the input volume. Every filter is con- volved with the input volume to generate a neuron activation map. The convolutional layer’s output volume is calculated by stacking 350 the activation maps of all filters along the depth dimension. Convo- lution operation output is calculated by convolving an input (I)with a number of filters as follows. xk=IWk+bk;k= 1;2;3; :::; F (10) where Fis the number of filters, xkis the output corresponding to thekthconvolution filter, Wkis the weights of the kthfilter, and bk355 is the kthbias. –Global Max Pooling Layer: A pooling layer is an additional layer that is inserted after the convolutional layer. Pooling layers give a method for downsampling feature maps by summarizing the existence of features in feature map patches. Maximum pooling, or 360 max pooling [38], is a pooling operation that calculates the maximum value in each patch of each feature map. The Global Max Pooling layer is another form of pooling layer where the pool size can be fixed to the same as the input size so that the maximum of the total input is calculated as the output value. 365 •Embedding Layer: An embedding layer is learnt alongside a neural network model on a particular natural language processing application, such as language modeling or text categorization. If an input sentence siis given, thewordsequencesofthissentence w1,w2,w3,...,wiisfedintoaword embedding layer to produce embedding vectors x1,x2,x3,...,xibefore being 370 sent to the next layer. The embedding layer is defined by an embedding matrix ERKX jVj, where Kindicates the embedding dimension and jVj, the vocabulary size. •Pre-trained Word Embeddings: Pre-trained Word Embeddings are embeddings that are learnt in one task and then applied to solve another 375 17 related problem. In this paper, we have used the following pre-trained word embeddings available in Bengali. –GloVe[39] creates the feature vector based on global and local word counts, word-word co-occurrence, and local context with the center word. GloVe’s semantic and syntactic features can be extracted more 380 effectively. However, owing to matrix factorization, it takes a long time. In our task, we have used Bengali-GloVe3. –Word2Vec [40] is a prediction-based embedding approach that gen- erates an embedding vector from the center word to the context word or vice versa. In this paper, we have used Bengali-Word2Vec3. 385 –BPEmb [41] model based on Byte-Pair encoding, which gives a col- lection of pre-trained subword embedding models for 275 languages including Bengali4. –FastText [42] is a prediction-based embedding approach that con- veys sub-word information. We have used FastText created for 157 390 languages including Bengali5. 4.4.2.2 Sarcasm Detection Models Architecture .A detailed description of all deep learning models for sarcasm detection is provided below. a.LSTM: A single hidden LSTM layer is followed by a typical feedforward 395 output layer in the original LSTM model. After preprocessing, texts are passed through a tokenizer and a one-hot encoding vector of length 100 is generated because Facebook comments are usually long. These vectors are then fed into the embedding layer. The output of the embedding layer is fed into the LSTM layer. Finally, a dense layer is added with a sigmoid 400 3https://bnlp.readthedocs.io/en/latest/ 4https://bpemb.h-its.org/ 5https://fasttext.cc/docs/en/crawl-vectors.html 18 activation function. The number of nodes in the dense layer is two because of the binary classification task. The vocabulary size is 10000. b.LSTM + CNN: This model is the combination of the LSTM and CNN models. The architecture of LSTM and the input of the embedding layer are the same as the model mentioned in subsection 4.4.2.2(a). After the 405 embedding layer, a 1D convolutional layer with 100 filters and kernel size 4 is added to speed up the longer training time. Next, a global max pooling layerwithpoolsize5isusedtoextractthemaximumvaluefromeachfilter and the output is the input of the LSTM layer. This vector is directly passed to a dense layer which is the output layer with sigmoid activation 410 function and the number of output nodes is the number of labels in the dataset. c.LSTM + CNN + Pre-trained Word Embedding: Thearchitecture of this model is the same as the model mentioned in subsection 4.4.2.2(b). The weights of the embedding layer are initialized with the weights of pre- 415 trainedwordembedding. Adropoutlayer,thenadenselayerisaddedafter the embedding layer. After that, a 1D convolutional layer and a global max-pooling layer are added and the output is the input of the LSTM layer. This vector is directly passed to the output layer with sigmoid activation function as mentioned in subsection 4.4.2.2(b). 420 d.Stacked LSTM + CNN + Pre-trained Word Embedding: This modelisthecombinationofstackedLSTMandCNNmodels. TheStacked LSTM is a variation of the LSTM model that includes multiple hidden LSTM layers, each of which contains multiple memory cells. The architec- ture of CNN is the same as the model mentioned in subsection 4.4.2.2(c). 425 The output of the LSTM with 1D convolution is passed to another LSTM layer before being used as the input of a dense layer. The obtained vector is directly passed to a dense layer which is the output layer with sigmoid activationfunctionandthenumberofoutputnodesisthenumberoflabels in the dataset. 430 e.Bi-LSTM + CNN + Pre-trained Word Embedding: A bidirec- 19 tionalLSTM [43], oftenknownasbiLSTM,isasequenceprocessingmodel that consists of two LSTMs, one of which takes the input forward and the other backward. BiLSTMs effectively improve the quantity of data avail- able to the network, allowing the algorithm to understand the context 435 better (knowing what words immediately follow and precede a word in a sentence). The architecture of the model remains the same as the model mentioned in subsection 4.4.2.2(c). Only the LSTM layer is replaced with the BiLSTM layer. f.Bi-LSTM + Pre-trained Word Embedding: Thearchitectureofthe 440 model remains the same as the model mentioned in subsection 4.4.2.2(e) by dropping the CNN portion of the model. Deep learning models require a longer training time as these process input sequence token by token. As a result, we will monitor how the Ben-Sarc dataset performs on transfer learning in the later to save the computational cost.445 4.4.3.Transfer Learning Transfer learning is a machine learning procedure in which the starting point of a new task is an already produced model for similar tasks [44]. Transfer learn- ing approaches have been effectively used for speech recognition, document cat- egorization, and sentiment analysis in natural language processing [45]. Figure450 5 represents an illustration of the transfer learning approach. Figure 5: Illustration of Transfer Learning [44] In transfer learning, we can utilize pre-trained source models available for developing new models. A plethora of transformer-based models for various 20 NLP tasks has recently emerged. The significant improvement of transformer-455 based models over RNN-based models is that these models accept the complete sequence as input all at once instead of analyzing an input sequence token by token. For this reason, we have utilized BERT. But BERT is a large neural network architecture with a massive number of parameters that may vary from 100millionto over300million. As aresult, traininga BERTmodelfrom scratch460 on a limited dataset would result in overfitting. Moreover, the computational cost of pre-training a BERT model is very high. As a result, as a starting point, it is preferable to employ a pre-trained BERT model that was trained on a large dataset. We can then further train the model using our relatively small dataset for fine-tuning. This approach is called fine-tuning. That is why we have used465 transfer learning approaches. 4.4.3.1 BERT .Bidirectional Encoder Representations from Transformers(BERT) [46] is one of the most prevalent transformer-based models which is used for pre-training a transformer [47]. BERT generates deep bidirectional word representations in470 unlabeled text based on the words’ contextual relationships to their surround- ings. Depending on its vocabulary, it generates word-piece embeddings. BERT pre-training is carried out using a masked language model (MLM), which ran- domly masks words that the model will estimate and compute the loss, and a next sentence prediction task, in which the model can predict the next sentence475 from the present sentence. Leta1,a2,....,a6besentencewords. a5israndomlymaskedwiththe [MASK ] token. The output of the sentence’s words is thus b1,b2,...,b6. The outputs are then routed through a block that includes two Fully Connected Layers, a GELU480 layer, and a normalization layer. The sentence and the anticipated value of the masked token are both outputs of the block. Three pre-trained BERT-based transformer language models are used as source models available in Hugging 21 Face Transformer’s library6as these are mostly used in downstream works like text classification. The transformer language models are-485 •Bangla BERT(base) [48], a pre-trained Bengali language model based onmasklanguagemodelingthathasbeenpre-trainedonBengaliWikipedia Dump dataset7and a large Bengali corpus taken from Open Super-large Crawled Aggregated coRpus (OSCAR)8. The model follows the bert-base- uncased model architecture that means it has 12 layers, 768 hidden layers, 490 12 heads, and 110M parameters. •Indic-Transformers Bengali BERT [49], a BERT language model that has been pre-trained on about 3 GB of monolingual training corpus, majorly taken from OSCAR8. It has achieved state-of-the-art performance on the Bengali language for the text classification tasks. 495 •Multilingual BERT(m-BERT) [46], a pre-trained model on 102 lan- guages with the largest Wikipedia including Bengali. We have used the model ’bert-base-multilingual-uncased’. It has 12 layers with 768 hidden layers, 12 multi-headed attention layers, and 110M parameters. 4.4.3.2 Architecture of Our Model500 .The detailed architecture of our model is shown in fig 6. The fine-tuning strategies of our new models can be divided as follows: a.Selecting Pre-trained Source Model: As explained earlier, BERT- based transformer models mentioned in section 4.4.3.1 are taken to this experimentaspre-trainedsourcemodelstoobservehowthesemodelswork 505 on transfer learning. b.Freezing the Entire Architecture of Source Model: Before fine- tuning, all the layers of each pre-trained language model are kept frozen 6https://huggingface.co/ 7https://dumps.wikimedia.org/bnwiki/latest/ 8https://oscar-corpus.com/ 22 by freezing BERT’s weight. This process prevents updating of model weights during fine-tuning. 510 c.Attaching Our Own Neural Networks Architecture: A different number of dense layers with different activations mentioned in 5.4.3 and softmax as output layer of our own are appended to the architecture to train this new model. Softmax can be expressed as- ai=ezi Pc k=1ezkwherecX i=1ai= 1 (11) The weights of the appended layers are updated during model training. 515 Different optimizers and learning rates are experimented with to get the optimized hyperparameter which is explained in section 5.4.3. Figure 6: Model Architecture of Sarcasm Detection for Transfer Learning 4.5. Phase V - Evaluation In the last phase, we have measured the performance of all models of phase IV. Then, the achieved results are compared and the best-performed mode is520 reported. The details of measuring the performance of the models are discussed briefly in the experiment section. 23 5. EXPERIMENTAL EVALUATION 5.1. Experimental Setup Python Keras framework with Tensorflow is used as a background to imple- 525 ment all deep learning models and Pytorch library is used for transfer learning models for training, tuning, and testing. Experimental evaluation was con- ducted on a machine with an Intel Core i5 processor with 2.71GHz clock speed and 4GB RAM. Tensorflow based experiments can utilize GPU instructions. Google Colaboratory has been used for developing all the models described in530 this paper in later sections as we have used Python language. 5.2. Experiments Our experiments are categorized into three parts. Experiment I is concerned with the experiments on traditional classifiers. Experiment II is focused on the experiments on deep learning classifiers and experiment III is reported on the535 experiments on transfer learning approaches. To judge the effectiveness of the models, accuracy, precision, recall, and f1-score measurements are taken into account. After hyperparameter tuning, the variation of results has been obtained for each model. So, only the better-540 performed model from each experiment has been taken in the Result Analysis section. 5.2.1. Experiment I In this experiment, the performance of traditional machine learning classi- fiers mentioned in 4.4.1 for the Ben-Sarc dataset has been evaluated. For this545 experiment, 20% of our data is used for testing purposes. The rest is used for training. A full overview of the performance with necessary evaluation met- rics for the experiment I has been demonstrated in the table 5. The process of choosing hyperparameters for the experiment I has been discussed in section 5.4.1. From table 5, it is shown that the MNB classifier has achieved the highest550 accuracy for the bigram technique with both 5-fold and 10-fold cross-validation 24 Table 5: Performance(in %) of 5-Fold and 10-Fold Cross-Validation for Experiment I Classifier5-Fold 10-Fold Technique Accuracy Precision RecallF1 ScoreTechnique Accuracy Precision RecallF1 Score LR bigram 71.80 71.01 73.71 72.33 bigram 72.06 71.29 73.88 72.56 DT bigram 61.92 61.58 63.44 62.49 bigram 62.42 62.17 63.37 62.76 RF trigam 70.52 69.93 72.03 70.96 trigram 70.89 70.11 72.89 71.47 MNB bigram 72.01 72.54 70.81 71.67 bigram 72.36 72.92 71.14 72.02 KNN unigram 64.52 67.72 55.50 61.00 unigram 64.67 67.91 55.63 61.14 Linear SVMunigram 71.03 69.04 76.28 72.47 unigram 71.29 69.47 75.99 72.58 Kernel SVMunigram 71.51 70.15 74.95 72.46 unigram 71.84 70.59 74.90 72.67 among all traditional classifiers as it works well with high dimensional text data by taking the advantage of probabilistic algorithm. It is 72.01% for 5 fold cross-validation and 72.36% for 10 fold cross-validation. 5.2.2. Experiment II555 In this experiment, the performance of different deep learning classifiers mentioned in subsection 4.4.2 for the Ben-Sarc dataset has been evaluated. For this experiment, 20% of our data is used for testing purposes. The rest is further divided into 60% for training and 20% for the validation set. A full overview of the performance with necessary evaluation metrics for experiment II has been560 demonstrated in the table 7. For LSTM models, LSTM units have been taken as 100 and for stacked LSTM, LSTM units have been taken as 128 and 64. 100 epochs have been used for all deep learning models. Though the epoch number was set as 100, it was stopped earlier due to early stopping criteria for monitoring 2 epochs with no improvement of the model’s performance. Binary565 cross-entropy is used as the loss function for all cases as the task is a binary classification problem. For all cases, dropout probability, and recurrent dropout probability have been set as 0.2. The hyperparameter setting is shown in the table 6. The procedure for picking hyperparameters for experiment II is covered in section 5.4.2.570 From table 7, it is clear that LSTM without pre-trained word embedding has achieved the highest accuracy of 72.48%. When an extra CNN and max-pooling layerisaddedtothismodel, theperformanceofthemodelhasdecreasedslightly. Then, the performance of this LSTM+CNN model has decreased more after 25 Table 6: Hyperparameter Setting of Each Best Performed Model for Experiment II ModelDense Layer SizeBatch SizeActivation in Hidden LayersOptimizerLearning Rate LSTM 1000 16 tanh Nadam 0.0001 LSTM+CNN 1000 16 tanh Nadam 0.0001 LSTM+CNN+GloVe 1000 16 tanh Nadam 0.00001 Stacked LSTM+CNN+GloVe 1000 16 tanh Nadam 0.00001 BiLSTM+CNN+GloVe 1000 16 tanh Nadam 0.00001 BiLSTM+GloVe 1000 16tanh ReLUNadam 0.00001 Table 7: Performance(in %) of Each Model for Best Setting of Experiment II Model Accuracy Precision Recall F1 Score LSTM 72.48 72.52 72.53 72.53 LSTM+CNN 69.40 69.31 69.49 69.39 LSTM+CNN+GloVe 66.09 65.99 65.92 65.95 Stacked LSTM+CNN+GloVe 65.91 65.80 65.88 65.84 BiLSTM+CNN+GloVe 65.58 65.67 65.84 65.75 BiLSTM+GloVe 61.64 62.04 58.66 60.24 using pre-trained word embedding. After that, the performance of other models575 has decreased gradually by adding or removing certain parts of the model. The reason for decreasing the models’ accuracy using pre-trained word embedding is that pre-trained word embeddings are mainly trained on a large dataset like Wikipedia where most of the language is very formal. But in the Ben-Sarc dataset, 87.65% of text are written in dialect, manipulating phrases, sentences,580 and spelling mistakes which determines the text as sarcastic as mentioned in section 3.5. 5.2.3. Experiment III In this experiment, the performance of transfer learning techniques men- tioned in 4.4.3 for the Ben-Sarc dataset has been evaluated. A full overview of585 the performance with necessary evaluation metrics for experiment III has been demonstrated in the table 9. The hyperparameter setting is shown in the table 8. The approach for optimizing hyperparameters for experiment III is outlined in section 5.4.3. Here, for all cases, Negative Log-Likelihood (NLL) [50] Loss has been used as 590 a loss function as it is the classic loss function used in any classification task [51]. 26 Table 8: Hyperparameter Setting of Each Best Performed Model for Experiment III Pre- trained ModelNo of Hidden LayersNo of Nodes in Hidden LayersActivation in Hidden LayerDropout OptimizerLearning RateBatch SizeNo of Epoch M- BERT 7512, 256, 128, 64, 32, 16, 8tanh 0.1 Adam 0.0001 8 30 Indic- Transformers Bengali BERT7512, 256, 128, 64, 32, 16, 8tanh 0.1 Adam 0.0001 8 30 Bangla BERT7512, 256, 128, 64, 32, 16, 8tanh 0.1 Adam 0.0001 8 30 Table 9: Performance(in %) of Each Model for Best Setting of Experiment III Pre-trained ModelAccuracy Precision Recall F1 Score M- BERT 66.00 67.00 64.00 65.47 Indic-Transformers Bengali BERT 75.05 74.00 77.00 75.47 Bangla BERT68.00 69.00 64.00 66.41 Softmax activation has been used in the output layer for all cases. From table 9, it can be concluded that transfer learning approaches for Indic-Transformers Bengali BERT pre-trained model has obtained the highest accuracy among all pre-trained models. It has achieved 75.05% of accuracy by using seven hidden595 layers and the settings mentioned above. For the transfer learning approach, at first, we have taken the m-BERT transformer model as a pre-trained model. But the overall performance was not satisfactory. Then, we have replaced the pre-trained model with Indic-600 Transformers Bengali BERT keeping the same hyperparameter setting. Here, a significant increase in all the performance measurement metrics has been ob- served. Almost9%accuracyhasbeenincreasedbychangingonlythepre-trained model from m-BERT to Indic-Transformers Bengali BERT. Then, we have ex- perimented with another pre-trained model Bangla BERT, but the performance605 has degraded significantly. 27 Table 10: A Short Overview of the Best Performed Model(in %) from Each Experiment Experiment No Model Accuracy F1 Score Experiment I MNB 72.36 72.02 Experiment II LSTM 72.48 72.35 Experiment III Indic-Transformers Bengali BERT 75.05 75.47 5.3. Result Analysis The highest accuracy from each experiment has been shown briefly in table 10. From the table 10, it can be concluded that the performance of experiment III,whichmeansthetransferlearningapproachisslightlybetterthantraditional610 machine learning and deep learning classifiers. By using Indic-Transformers BengaliBERTasapre-trainedmodel, transferlearninghasobtainedthehighest accuracy of 75.05% for the Ben-Sarc dataset where LSTM without pre-trained word embeddings from deep learning classifiers and multinomial Naive Bayes from traditional classifiers achieved a maximum 72.48% and 72.36% accuracy615 respectively. 5.4. Hyperparameter Tuning Hyperparameter tuning is a necessary stage in each experiment to boost performance. The hyperparameter tuning has been carried out on all of our experimentsI,II,andIII.Athoroughexplanationofallofthemodelsisprovided620 in the following subsections. 5.4.1. For Experiment I For the experiment I, we have applied 5-Fold and 10-Fold cross-validation on seventraditionalclassifierslistedintable5. Unigram, bigram, andtrigramtech- niques have been applied for each classifier. Among them, the best results from625 each classifier have been demonstrated in table 5 for both cross-validation tech- niques. MNB classifier for 5-fold cross-validation has attained 71.85% accuracy and 72.13% for 10-fold cross-validation for the unigram technique. These are the second-best results for both 5-fold and 10-fold cross-validation, respectively. The performance of other classifiers cannot surpass these results.630 28 Table 11: Hyperparameters with Their Values Which Tuned across All Models for Experi- ment II Pre-trained Word EmbeddingDense Layer SizeBatch SizeActivation in Hidden LayersOptimizerLearning Rate GloVe, Word2Vec, BPEmb, fastText, and without Pre-trained word embedding1000, 2000 8, 16, 32 tanh, ReLU, Adam, Nadam0.01, 0.001, 0.0001, 0.00001 Table 12: Hyperparameters with Their Values Which Tuned across All Models for Experi- ment III Pre-trained Source ModelsBatch SizeNo of Hidden LayersNo of Nodes in Hidden LayersActivation in Hidden LayersNo of Dropout LayersDropout Proba- bilitiesOptimizerLearning RateNo of Epoch Bangla BERT, Indic- Transformers Bengali BERT, m-BERT4,8, 16, 321,2,3, 5,7,91024,512, 256,128, 64,32,16,8,4tanh, ReLU, ReLU6,sigmoid, selu1,2,3 0.1,0.2,0.5Adam, AdamW, Adamsx, SGD, RMSprop0.01, 0.001, 0.0001, 0.00001, 0.000510,20,30 5.4.2. For Experiment II For experiment II, all hyperparameters which have been tuned for several combinations across all models are mentioned in table 11. Among them, LSTM without pre-trained word embedding has achieved 71.37% on dense layer 1000, the number of LSTM layers 2, batch size 16, hidden layer activation tanh,635 Nadam optimizer with 0.0001 learning rate. This is the second-highest accu- racy for experiment II. Other models with different combinations cannot obtain better this result. 5.4.3. For Experiment III For experiment III, all hyperparameters which have been tuned for several 640 combinations for all models are mentioned in table 12. The second highest accuracy from experiment III is 74.00% which is achieved from two settings - first one : 3 hidden layers with 512,256, 128 hidden layer nodes, tanh hidden layeractivation, 0.1dropout, SGDoptimizerwith0.01learningrate, batchsize8 with 30 epochs and the second one: 4 hidden layers with 512,256, 128, 64 hidden645 layer nodes, sigmoid hidden layer activation, 0.2 dropout, Adam optimizer with 0.001 learning rate, batch size 4 with 30 epochs. 29 6. CONCLUSION AND FUTURE WORK In this paper, we have presented a benchmark dataset for Bengali sarcas- tic comments on Facebook to make an influence in one of the low resources650 languages named Bengali. Then we have demonstrated a thorough and compre- hensive strategy to utilize different models of machine learning, deep learning, and transfer learning. This is an attempt to make a contribution in the disci- pline of sentiment analysis on the Bengali language domain to achieve a boon in the branch of consumer research, opinion mining, branding, and so on. In the655 future, we wish to improve the quality of our work by increasing the size of our Ben-Sarc dataset. Besides, emoji and emoticons play a vital role to articulate the actual connotation of a comment on social media. So, we will consider emoji and emoticons along with the text.