SARCASMANIA: Sarcasm Exposed!
By: Ramsha Siddiqui
Tags: Machine Learning, Python Programming, NLP, Sentiment Analysis, Sarcasm

Hello there! This is an article featuring the release of Sarcasmania-API (now available for download), as well as a detailed description of the project, for your interest!

Here is the Application Download Link on Google Play:

Also on Amazon App-Store (FREE Checkout): https://amzn.to/30ByWMa

Application Overview

This is a Social Media Application where a nest of sarcastic tweets is measured on a polar scale of hilariousness and offensiveness by our users. The application allows our audience to post, view and impartially react to sarcastic tweets uploaded by our large pool of customers chronologically. Finally, an infinitely-running model on our server incorporates this input in a Man VS Machine probability-ratio (30/70) of humor and insult respectively. This application is part of a Natural Language Processing Research Project of improving Sarcasm Detection in Text through Continual Learning and Sentimental Analysis.

“Can I interest you in a sarcastic comment?” ~ Chandler Bing from F.R.I.E.N.D.S

Detailed Description

Below lies a thorough detail on the research work involved in this project, as well as its benefits, deficiencies and results.

Introduction

Social media platforms have gained immense popularity in recent years and because of their huge volume, they have now become one of the most effective data-sets for conducting research on any topic. For a business organization to be successful these days, social media is leveraged to tap into public opinion about their products and services for which they have dedicated teams for marketing, sales, and consumer assistance to maintain their social media presence. In addition to market analysis, one’s emotional reception of another’s hilarity is an essential factor used by various social-media applications, especially those uniting potential soulmates. Several psychologists would also agree that the actions and reactions surrounding such comments can be utilized in personality assessments, particularly those concerning anger management and depression.

Motivation

The extraction of research content related to a certain topic is done by using different tools on these platforms. Majority of the public makes the use of sarcasm to express their opinion on these sites especially when it’s a critical review. Since sarcasm is a nuanced form of language in which individuals state the opposite of what is implied, sarcasm detection has always been a challenging task for these tools because of this intentional ambiguity. But due to its excessive usage on social media, detection of sarcasm is important as it is beneficial to many NLP applications, such as sentiment analysis, opinion mining and advertising. Following these claims, we began this research on Sarcasm Detection and its improvement using sentimental analysis and user ratings.

Scope

The motive of our project was to develop a model that we will train for sarcastic, humorously sarcastic, insulting sarcastic and both (humorously and insultingly sarcastic) comments using machine learning algorithms for sarcasm, insult and humor. The intended audience for this project includes:

● Market Analysis Teams

● Psychologists

● Matchmakers

● Social Media Users

The timeline for this project was from Fall 2018 to Spring 2019 i-e the duration of one year to complete literature survey, python programming, android application development and server authentications, as shown:

A Snapshot of the Project Timeline from September 2018 to May 2019

Literature Survey

Listed below are the various research papers and survey articles thoroughly read and researched by members of our team — for their different approaches, functionalities, outputs and efficiencies. These research papers are categorized into three major types of machine-learning detection domains: sarcasm, humor and insult in text (natural language processing).

Research on Sarcasm-Detection: Please refer to the table below.

A brief summary of a few selective Sarcasm-Detection Publications

Research on Humor-Detection: Please refer to the table below.

A brief summary of a few selective Humor-Detection Publications

Research on Insult-Detection: Please refer to the table below.

A brief summary of a few selective Insult-Detection Publications

Implementation

Our implementation is divided into two main modules, described below:

Calculation of Scores using Machine Learning Algorithms: This module of our system is responsible for determining the scores for sarcasm, insult and humor using machine learning algorithms.

Integration of User Feedback into the Predetermined Scores: After the first module has assigned certain scores for sarcasm, insult and humor to the input tweet, the second module will then be responsible for integrating user feedback into the original scores. For this purpose, we will assign specific weightage to user input and keep on adding average scores into the original assigned scores based on 70/30 Machine-User Ratio.

Sarcasm Detector

The first machine learning algorithm will identify the sarcasm in the input tweet and assign it score based on its sarcasm level. The tweet will be further processed if it exceeds a certain threshold level of our system to eliminate irrelevant data. A hybrid model that leverages algorithms for both types of sarcasm has been used based on the two-types of sarcasm given as:

Context-based Sarcasm:

● User Profiling: The motivation for including this is that users tend to be sarcastic / non-sarcastic across forums and their behavioral features can be utilized as contextual information. We generate a user-embedding for each user based on two user-traits:

○ Stylometric features: Generated using Paragraph-Vector algorithm.

○ Personality Traits: A CNN model trained on a benchmark dataset.

● Discourse Modeling: The motivation for including this is that certain forum-topics correlate to sarcastic inclination of the posts — e.g. Politics more prone to sarcasm than discussion on natural disasters. Similar to stylometric features, Paragraph-Vector model is applied on accumulated posts from each forum, based on the following:

○ Contextual and Topical information.

○ Document modeling of the discourse of discussion. Content-based Sarcasm

We Capture lexical cues present in sentences that help detect sarcasm. Sentential representations are extracted from the target comment by using a multi-layered Convolutional Nueral Network (CNN).

Sarcasm Score of an Input Tweet

Insult Detector

A toxic comment classifier has been implemented using Neural Networks. A breakdown of the CNN is given as follows:

Embedding Model: Pre-trained relationships between words, which is a matrix 300 x 400,000. Index2word: A dictionary containing key-value pairs, the key being the word as a string and value being the integer representing the word. Note, these integers correspond with the index in the Embedding Model. Word2idx: A list containing all the words. The index corresponds to the word’s position in the word embeddings. Essentially, the reverse of the index2word.

The process of creating the model starts with loading the word-embeddings and the dictionaries (index2word and word2idx). Next, we load the labelled toxic comments and convert these data-frames into the sequences. Next, we create a tokenizer object and fit it to the sequences as well as pad all the sequences, so they are the same size. After that, we look up the word-embedding vector for each unique word in sequences and store the wordembedding vector in the embeddings’ matrix. If the word is not found in the embeddings, then leave the index all zeroes. Also, limit the embedding-matrix to the 20,000 most used words. Once we’re done, we create a Keras Embedding Layer from the embeddings’ matrix and split the data for training and validation. Finally, we create a CNN model with rectifier activation and a binary cross-entropy loss. This model is now fully capable of making predictions about a comment’s toxicity levels, obscenity, threat-level, insult-factor and identity-hate affirmation.

Continual Learning: Continual Learning is an unexplored region of machine learning even in 2019. The core principle that it operates on is that a machine’s most important knowledge is not forgotten while at the same time, new information is learned sequentially. Its properties are:

Multi-Task Learning: Tasks arrive sequentially.

Tasks arrive sequentially. Auto-Updating: The model can update itself on new tasks and data distributions, without forgetting previous knowledge.

Now let’s try and understand it in more detail as to why it exists and why its integration is necessary for this project.

Problems: The two main obvious problems that exist for all machine learning models that need to learn new information are:

Catastrophic Forgetting: Machines forget old data and only factor in the new data with its labels. Degree of Dissimilarity between tasks: If the two tasks that are being learnt by the model are quite different, this would also ultimately result in failure.

Solutions: A few solutions that come to mind based on these problems are:

Complete re-training on the whole data combined: This approach is affective but naïve as its time-complexity would increase manifold. Retain previous knowledge: A method of retaining previous knowledge includes saving old parameters for reuse. Utilize previous knowledge: The technique through which previous knowledge is utilized is called Transfer Learning. Learn new knowledge: The technique of learning new dataset is called Incremental Learning by dynamically increasing parameters.

Knowledge Distillation: In this Continual Learning technique, we work on the model response level (output) instead of editing its parameters.

We have to keep the new response level really close to the original one and also keep the new parameters close to the old ones in order to maintain its performance level. The formula for loss is defined as:

Loss( λ) = Old Output - New Output

Multi-task learning: It works on the principle of ‘Learning without Forgetting’. Here, we first learn the initial task, then run it on new data and record responses and finally expand the model with a new classification layer that extends the output accordingly. In order to do this, we find a good value of λ. If λ is low, we ignore the new data, otherwise, we learn from new data. This means that this algorithm is highly dependent on the connectivity of the tasks — if similar, they only move an inch from their old values, otherwise, they fight for power!

It works on the principle of ‘Learning without Forgetting’. Here, we first learn the initial task, then run it on new data and record responses and finally expand the model with a new classification layer that extends the output accordingly. In order to do this, we find a good value of λ. If λ is low, we ignore the new data, otherwise, we learn from new data. This means that this algorithm is highly dependent on the connectivity of the tasks — if similar, they only move an inch from their old values, otherwise, they fight for power! Incremental Learning of Object Detectors: In this approach we work through exposing old responses on old classes and using entropy loss on new classes. In this case, we don’t add a new classification layer, but only add a few neurons in the output layer.

Application: Knowledge Distillation was ultimately used to train our model on user input.

In order to incorporate User Feedback from the audience testing our android application in Beta Phase, we required a learning algorithm that can learn new data (related to the old data) and improve its efficiency accordingly. More importance would be given to the initial task (training dataset) and in order to incorporate loss — a few dropout layers have been added into our original insult detection model.

Accuracy: Its test accuracy so far is 95.38% but is subject to change as more and more user data arrives.

Insult Score of an Input Tweet

Humor Detector

The humor detection unit will work in parallel with the insult detector to determine the level of humor present in the input tweet and assign it a humor score. Shallow Learning Method of SVMs (Support Vector Machines) with Linear kernel is being used with Yelp reviews as training data to the model.

Incremental Learning: The humor detection algorithm that was previously being used was an SVM with a linear kernel trained on Yelp reviews for humor detection. There was no direct implementation available for implementing incremental learning on an SVM, so for this purpose SGD’s implementation provided by scikit learn is being used. The loss function that is being used is called ‘hinge’ which outputs a linear SVM same as the one used in the previous implementation of humor detection. The model is being trained using Yelp reviews which were used in the previous implementation as well, but the model is not being completely fit on the reviews, SGD’s partial_fit function is being used so that we can incrementally make the model learn sarcastic humor over time. The model will partial_fit again when new input is provided to it through our application.

Humor Score of an Input Tweet

Merger Algorithm

Lastly in this module, the merger algorithm will compare humor and insult scores of the input tweet to evaluate a combined score for it which will range between the two extremes of insult and humor on a scroll bar. For this purpose, a mathematical formula with certain weightages will be used to merge the two scores.

Sarcasm, Humor and Insult Score of an Input Tweet

Results

The application is under testing by selective users to train the Machine Learning models for most-accurate predictions of sarcasm, humor and insult inside a given tweet. Meanwhile, the implementation of all that is mentioned above can be found here:

Multi-Feature Detection Model

Using our self-labelled data-set, merged with user responses from Continual Learning, we were able to produce a Multi-Feature Sarcasm, Humor and Insult Detection Model, whose code is linked below:

As you can see, the model trained with an accuracy of 99.08% — and the correlation of the data is visible from the dataset found on Kaggle here:

At the end the Probabilities that prove correlation between the three sentiments are deduced from the data below:

P(Sarcasm) = 18488/39780 = 0.46 P(Humor) = 19643/39780 = 0.49 P(Insult) = 19358/39780 = 0.48 P(Sarcasm|Humor) = 9037/39780 ÷ 0.49 = 0.44 P(Sarcasm|Insult) = 8694/39780 ÷ 0.48 = 0.4375 P(Sarcasm|Humor,Insult) = 4293/39780 ÷ .46*.49*.48 = 0.92

This concludes our research proving the usefulness of the relation between sarcasm, humor and insult — that can now be used for proper detection of Sarcasm in text.

Future Goals

We are on our way to get a research publication out of this research from the IEEE Board this Summer in 2019. Furthermore, this research could be extended to find the correlation of Humor with Insult directly — as those two often overlap as well. We are also going to submit this as an idea for Facebook’s Fake-News Challenge: Based on the comments on a post, or the post’s status/content — we can use sarcasm detection to figure out if the author is joking or serious in News Centeres such as 8Fact and 9Gag.

Team Members:

Ramsha Siddiqui, Maryam Nadeem, Mahnoor Khan and Neha Akram

Project Adviser:

Dr. Maryam Bashir

Center for Publication:

Department of Computer Science, National University of Computer and Emerging Sciences, Lahore, Pakistan (Final Year Projects 2018)

Please comment below with your feedback about the Application, and make sure to download and use it atleast once! Thank you ❤ …