Twitter U.S. Airline Sentiment Analysis using Keras and RNNs
By: Francesca Lim
Tags: Machine Learning, Sentiment Analysis, Word Embeddings, Rnn, Keras

Introduction

Sentiment analysis is a type of natural language processing problem that determines the sentiment or emotion of a piece of text. For example, an algorithm could be constructed to classify whether a product’s review was positive, neutral, or negative. Natural language processing (NLP) is a field of artificial intelligence that involves computers understanding and processing human language. The goal is often to derive meaning from text.

There are many different applications of NLP. These include language translation, text generation, and spell/grammar check. Additionally, there are numerous techniques that can be applied to analyze text and extract meaning. For this mini-project, we will focus on building a recurrent neural network (RNN) to classify the sentiment of tweets about airlines using Keras and a pretrained word embedding.

Data

Our dataset is called “Twitter US Airline Sentiment” which was downloaded from Kaggle as a csv file. Its original source was from Crowdflower’s Data for Everyone library. Tweets were scraped from Twitter in February 2015 about each major US airline. Contributors then classified each tweet as either “positive”, “neutral”, or “negative” and cited the reason for a negative classification as well as a confidence score for the assigned label.

There are 14,640 rows and 15 columns. The included features are: tweet id, sentiment, sentiment confidence score, negative reason, negative reason confidence, airline, sentiment gold, name, retweet count, tweet text, tweet coordinates, time of tweet, date of tweet, tweet location, and user time zone.

First 5 rows of the dataset

Background

Recurrent Neural Network (RNN)

Unlike feedforward networks that process each input individually and independently, a RNN creates loops between each node in the neural network. This makes it particularly good for sequential data, such as text. It is able to process sequences and create a state which contains information about what the network has seen so far. This is why RNNs are useful for natural language processing, because sentences are decoded word-by-word while keeping memory of the words that came beforehand to give better context for understanding. A RNN allows information from a previous output to be fed as input into the current state. Simply put, we can use previous information to help make a current decision.

Long Short-Term Memory (LSTM)

Simple RNNs suffer from the vanishing gradient problem which occurs when information from earlier layers disappear as the network becomes deeper. A LSTM algorithm was created to avoid this problem by allowing the neural network to carry information across multiple time steps. This means it can save important information for later use, preventing gradients from vanishing during the process. Additionally, a LSTM cell can determine what information to remove as well. Therefore, it can learn to recognize an important input and store it for the future while removing unnecessary information.

Gated Recurrent Unit (GRU)

A GRU layer was also created in response to the vanishing gradient problem. It is a simplified version of a LSTM cell but performs just as well.

Bidirectional RNNs

In general, RNNs tend to be order or time dependent. They process the time steps in a sequential, unidirectional order. On the other hand, a bidirectional RNN is able to process a sequence in both directions which means it may be able to pick up patterns that would not be noticed using a unidirectional model. Therefore, this type of model is able to improve performance on problems that have a chronological order.

Word Embeddings

In NLP, textual data must be represented in a way that computers can work with. We will focus on word embeddings which is a representation of text where similar words have a similar representation. One model of word embedding is word2vec which takes a large corpus of text and outputs a vector space where each unique word has its own corresponding vector. In this space, words with similar meanings are located close to one another.

Another popular model is the Global Vectors for Word Representation (GloVe) which is an extension of word2vec. It generally allows for better word embeddings by creating a word-context matrix. Basically, it creates a measure to indicate that certain words are more likely to be seen in context of others. For example, the word “chip” is likely to be seen in the context of “potato” but not with “cloud”. Its developers created the embeddings using English words obtained from Wikipedia and Common Crawl data.

We used a pre-trained word embedding, because we believe GloVe generalize with our data. The embedding space created by GloVe likely contains all the words we will encounter in our tweets, so we can use these vector representations instead of creating our own from a much more limited vocabulary set.

Purpose

The goal of this mini-project is to classify airline tweets as positive, neutral, or negative using a pre-trained word embedding from GloVe and a RNN.

Exploratory Data Analysis

First, we can look at the counts for each sentiment label. We note that there are more negative tweets than both positive and neutral combined. We can potentially infer that people are more likely to be vocal on social media platforms, such as Twitter, when something goes wrong with their flight as opposed to when nothing unexpected happens.

Number of Tweets in each Sentiment Group

To better understand the difference in sentiments for positive, neutral, and negative tweets, we can create a word cloud for each category. Before we create this word cloud, we must clean the tweets by removing twitter handles, special characters, numbers, punctuation, and small words that are 3 characters or less. This allows us to see words that are representative for each sentiment as opposed to words that are likely seen across each label. We look at the top 75 most used words in each sentiment group.