TensorFlow: Sarcasm Detection in 20 mins | by Aaditya Bhat | Towards Data Science

This is a quick Natural Language Processing (NLP) tutorial in TensorFlow. NLP is a sub-field of artificial intelligence concerned with understanding, interpreting and manipulating human language by computers. TensorFlow is an open source machine learning library by Google. This tutorial assumes intermediate understanding of Python, basic understanding of Machine Learning, and requires Python 3 with TensorFlow 2.x. At the end of this tutorial, you will be able to train your own Sarcasm Detection model.
We want to build a neural net classifier to detect sarcasm in text. For this, we will be using ‘Sarcasm in News Headlines Dataset by Rishabh Misra’. Following code parses the dataset -
# Importing required librariesimport jsonimport tensorflow as tfimport requestsimport numpy as npimport pandas as pdfrom tensorflow.keras.preprocessing.text import Tokenizerfrom tensorflow.keras.preprocessing.sequence import pad_sequences# Get the datasetsrcsm_json = requests.get('https://storage.googleapis.com/laurencemoroney-blog.appspot.com/sarcasm.json')# Inspecting the data, print 450 charactersprint(srcsm_json.text[0:450])>>> [>>> {"article_link": "https://www.huffingtonpost.com/entry/versace- >>> black-code_us_5861fbefe4b0de3a08f600d5", "headline": "former >>> versace store clerk sues over secret 'black code' for minority >>> shoppers", "is_sarcastic": 0},>>> {"article_link": "https://www.huffingtonpost.com/entry/roseanne- >>> revival-review_us_5ab3a497e4b054d118e04365", "headline": "the >>>'roseanne' revival catches up to our thorny political mood, for >>> better and worse", "is_sarcastic": 0},# Separate the json into sentences and labelssentences = []labels = []for item in srcsm_json.json():    sentences.append(item['headline'])    labels.append(item['is_sarcastic'])print(pd.DataFrame({'sentence' : sentences[0:10], 'label':labels[0:10]}))>>>                                            sentence  label>>> 0  former versace store clerk sues over secret 'b...      0>>> 1  the 'roseanne' revival catches up to our thorn...      0>>> 2  mom starting to fear son's web series closest ...      1>>> 3  boehner just wants wife to listen, not come up...      1>>> 4  j.k. rowling wishes snape happy birthday in th...      0>>> 5                        advancing the world's women      0>>> 6     the fascinating case for eating lab-grown meat      0>>> 7  this ceo will send your kids to school, if you...      0>>> 8  top snake handler leaves sinking huckabee camp...      1>>> 9  friday's morning email: inside trump's presser...      0
We have our initial dataset of sentences and corresponding label 1 or 0 indicating whether the sentence was sarcastic or not. The sarcasm classifier will use sentence as input and will predict the label. Before we start training the classifier, we need to represent the input data, i.e. sentences in a way that a computer can process them.
This is achieved by a combination of tokenization and sequencing. Tokenization is a task of chopping a string into pieces, called as tokens, while throwing away certain characters like punctuation marks. Tokenization assigns a number to each unique word. Sequencing builds on top of tokenization, by representing sentences as a sequence of numbers.
In practice, our classifier might encounter words it hasn’t seen in the training data, in such cases the classifier will ignore those words altogether, resulting in some information loss. To minimize the information loss, during tokenization we assign a token to represent all the unseen (out of vocabulary) words. For the neural net to handle sentences of variable lengths we use padding. In padding we set a maximum allowed length for the sentences, all the sentences shorter than the maximum length would be padded to match the maximum length. Conversely, all the sentences longer than the maximum length would be truncated to match the maximum length. Following code performs tokenization and sequencing -
# Splitting the dataset into Train and Testtraining_size = round(len(sentences) * .75)training_sentences = sentences[0:training_size]testing_sentences = sentences[training_size:]training_labels = labels[0:training_size]testing_labels = labels[training_size:]# Setting tokenizer propertiesvocab_size = 10000oov_tok = "<oov>"# Fit the tokenizer on Training datatokenizer = Tokenizer(num_words=vocab_size, oov_token=oov_tok)tokenizer.fit_on_texts(training_sentences)word_index = tokenizer.word_index# Setting the padding propertiesmax_length = 100trunc_type='post'padding_type='post'# Creating padded sequences from train and test datatraining_sequences = tokenizer.texts_to_sequences(training_sentences)training_padded = pad_sequences(training_sequences, maxlen=max_length, padding=padding_type, truncating=trunc_type)testing_sequences = tokenizer.texts_to_sequences(testing_sentences)testing_padded = pad_sequences(testing_sequences, maxlen=max_length, padding=padding_type, truncating=trunc_type)
At this stage, we are ready to train a neural net on our data, we have sequences of numbers representing the sentences. To determine if something is sarcastic from the sequences, we use the concept of embedding. Let’s say we represent words with vectors, for example, the word ‘good’ is represented by (1,0) and the word ‘bad’ which is opposite of good is represented by (-1,0). Words which mean something positive will be close to (1,0), while the words which mean something negative will be close to (-1,0). If we extend this framework to multiple dimensions, a sentence can be represented by the sum of vectors of the words in the sentence. While the neural net classifier learns to detect sarcasm, it learns the direction in these multi-dimensions. Sarcastic sentences will have a strong component in sarcastic direction, while other sentences will have a strong component in not sarcastic direction. As we train the neural net classifier with more data, these directions might change. When we have a fully trained neural net classifier, it can sum the vectors for all the words in sentence and predict whether the sentence is sarcastic or not.
The final neural net consists of a top layer of embedding which learns the direction of each word, the next layer is Global Average Pooling which adds up the vectors, the next layer is a deep neural network and the final layer is a sigmoid layer which returns the probability of the sentence being sarcastic. Following code trains the neural net -
# Setting the model parametersembedding_dim = 16model = tf.keras.Sequential([    tf.keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length),    tf.keras.layers.GlobalAveragePooling1D(),    tf.keras.layers.Dense(24, activation='relu'),    tf.keras.layers.Dense(1, activation='sigmoid')])model.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])model.summary()>>> Model: "sequential_4">>> ________________________________________________________________>>> Layer (type)                 Output Shape              Param #   >>> ================================================================>>> embedding_4 (Embedding)      (None, 100, 16)           160000    >>> ________________________________________________________________>>> global_average_pooling1d_4 ( (None, 16)                0         >>> ________________________________________________________________>>> dense_8 (Dense)              (None, 24)                408       >>> ________________________________________________________________>>> dense_9 (Dense)              (None, 1)                 25        >>> ================================================================>>> Total params: 160,433>>> Trainable params: 160,433>>> Non-trainable params: 0>>> ________________________________________________________________# Converting the lists to numpy arrays for Tensorflow 2.xtraining_padded = np.array(training_padded)training_labels = np.array(training_labels)testing_padded = np.array(testing_padded)testing_labels = np.array(testing_labels)# Training the modelnum_epochs = 30history = model.fit(training_padded, training_labels, epochs=num_epochs, validation_data=(testing_padded, testing_labels), verbose=2)>>> Train on 20032 samples, validate on 6677 samples>>> Epoch 1/30>>> 20032/20032 - 2s - loss: 0.6742 - accuracy: 0.5669 - val_loss: >>> 0.6247 - val_accuracy: 0.6629>>> Epoch 2/30>>> 20032/20032 - 1s - loss: 0.4758 - accuracy: 0.8121 - val_loss:  >>> 0.4018 - val_accuracy: 0.8278>>> Epoch 3/30>>> 20032/20032 - 1s - loss: 0.3293 - accuracy: 0.8686 - val_loss: >>> 0.3708 - val_accuracy: 0.8327>>> ...>>> ...>>> Epoch 29/30>>> 20032/20032 - 1s - loss: 0.0310 - accuracy: 0.9920 - val_loss: >>> 0.9636 - val_accuracy: 0.8167>>> Epoch 30/30>>> 20032/20032 - 1s - loss: 0.0297 - accuracy: 0.9925 - val_loss: >>> 0.9431 - val_accuracy: 0.8131
The neural network was able to achieve 99% accuracy on the training dataset and 81% accuracy on the test dataset.
Let’s do a final sanity check by using the model to predict two new sentences.
sentence = ["Coworkers At Bathroom Sink Locked In Tense Standoff Over Who Going To Wash Hands Longer",             "Spiking U.S. coronavirus cases could force rationing decisions similar to those made in Italy, China."]sequences = tokenizer.texts_to_sequences(sentence)padded = pad_sequences(sequences, maxlen=max_length, padding=padding_type, truncating=trunc_type)print(model.predict(padded))>>>[[0.99985754]>>> [0.0037319 ]]
The model predicted a very high (almost 1) probability of ‘Coworkers At Bathroom Sink Locked In Tense Standoff Over Who Going To Wash Hands Longer’ being sarcastic, while a very low (almost 0) probability of ‘Spiking U.S. coronavirus cases could force rationing decisions similar to those made in Italy, China’ being sarcastic. We can conclude that our model is working as intended.
This tutorial was based on ‘Natural Language Processing (NLP Zero to Hero)’ series by TensorFlow on YouTube. The Jupyter notebook for this tutorial can be found here.