Reddit Sentiment Analysis With Python
By: Jason Lzp
Tags: Python Programming, Reddit, NLP, Sentiment Analysis, Data Science

They say that actions speak louder than words, but sometimes words can also mask emotions that are undetected by actions. In this guide, we would be looking at analysing the sentiment of comments from a Reddit post.

Reddit is a social news aggregation, web content rating, and discussion website that provides a ton of data for us to explore. By analysing the sentiment of words from specific posts, we can determine whether a piece of writing is positive, negative or neutral.

Sentiment analysis is a field of study that refers to natural language processing (NLP), and depending on the context, such analysis can be especially useful for certain sectors. For example, a company is trying to understand its customers' sentiment based on the reviews left on their products. Using the sentiment, the business can gather an overview of how customers react to products and develop future business strategies.

As with most inputs before analysis, cleaning would be needed. In our case, since we are dealing with qualitative data, the process might look a little different compared to quantitative data cleaning.

Let’s begin with an overview of the process for this project:

Establish a Reddit instance via API Obtain comments from a post on Reddit Preprocess the comments Apply a sentiment analyzer model (VADER) Obtain sentiment results and create some simple visualizations, e.g. word clouds

1. Establishing A Reddit Instance with Praw

Like my previous post, we would be working with a Python API Wrapper to obtain data in the form of comments for our analysis.

For Reddit, we would be utilizing the PRAW API Wrapper. Prerequisites for using the API Wrapper includes:

Python Knowledge

Reddit Knowledge (A basic understanding of how Reddit works)

Reddit Account (To access Reddit’s API)

Client ID & Client Secret (These two values are needed to access Reddit’s API as a script application)

User Agent (A unique identifier that helps Reddit determine the source of network requests)

For more information, you can visit https://praw.readthedocs.io/en/latest/.

Let’s being by creating an App via https://www.reddit.com/prefs/apps.

At the bottom of the page, locate and select the “are you a developer? create an app…” box. Fill in your API's name, select the script option, and fill up the redirect URL with “http://www.example.com/unused/redirect/uri”. Afterwhich proceed to create your app.

As Reddit’s API uses OAuth2, a Reddit connection must be established with a client ID and client secret. These values can be found after creating your app.

For our project, we would be utilising the following imports:

Should you encounter any errors while importing the libraries mentioned above, kindly install them via pip on your terminal.

Once done, we can begin by establishing a Reddit connection with our client ID and client secret.

2. Obtaining Comments From A Post

Next, we have to create a submission object before extracting the comments from a Reddit subreddit post.

The submission ID contains the ID for a specific post throughout Reddit. Since we are interested in obtaining the sentiments from an individual post's comments, we have chosen the top 5 “hot” posts popular throughout Reddit.

You can choose to display your desired number of posts that are “hot”, “new” or “top” from a specific subreddit as well.

You can also create a submission object of your desired post by taking the submission ID from the post directly from the address bar. For example:

In our case, let’s proceed with defining our submission object with Submission ID: kp57s7

Once we have our submission object, we would obtain all of the post's comments and store them into a list. This might take a while, depending on how much comments the post contains.

And our output, containing 4033 comments scraped from this post.

As you can see, the text contains a ton of words and might include irrelevant information such as links, emojis and words which we would have to remove before proceeding.

3. Preprocess The Comments

When dealing with raw or uncleaned data, we would first have to preprocess or “clean” it before modelling or in our case, to apply a sentiment analysis.

In our instance, we would include the following steps for our preprocessing:

Removing Emojis

Tokenizing, removing links etc.

Removing stopwords

Normalizing words via lemmatizing

We first begin by converting our list into a string object before passing them through each preprocessing process.

Next, we would remove Emojis that are in our string object.

Followed by tokenizing and cleaning the string via a regular expression tokenizer. Tokenizing breaks apart every word in the string into an individual word which would then carry it’s own “positive” or “negative” sentiment based on our sentiment analyzer later.

As you can see, some tokens are uppercased, and some are lowercased. To standardise this, we would convert all tokens to lower case and store it into a list.

Next, we would remove stop words in our list object.

Stop words are words that do not add much information to a sentence. As such, they can safely be ignored without sacrificing the meaning of the sentence. An example of stopwords contained in en_core_web_sm includes the following:

Here is our output after removing the stop words

Lastly, we would normalize the words via a process called: Lemmatizing or Stemming.

Both processes are used to trim words down to their root words. However, stemming might return a root word that is not an actual word whereas, lemmatizing returns a root word that is an actual language word.

As such, we would only be applying the lemmatizing process to our list object.

And the output

For comparison purposes, let’s run the stemming process on our list object to get an idea of the output.

Noticed that the words like ‘remember’ and ‘corporation’ in our lemmatized token output becomes ‘rememb’ or ‘corpor’ which is not an actual language word.

We will proceed by storing the lemmatized token into a new variable. For visualization purposes, we can see that the number of words pre and post-processing has reduced dramatically!

4. Apply A Sentiment Analyzer (VADER)

Once we obtain our cleaned output, we would calculate each tokenized word's polarity scores using the VADER (Valence Aware Dictionary for Sentiment Reasoning) model.

The polarity scores measure the positivity and negativity for each word. We are mostly interested in the compound score, which is normalized to be between -1 (most extreme negative sentiment) and +1 (most extreme positive sentiment). This provides a single unidimensional measure of sentiment for a given word.

We would then store this as a data frame object.

And the output

Next, we would add a new column ‘label’ which filters the words based on the compound score and our criteria for categorizing them. In our case, we used a compound score of +- 0.10 to discern between positive, negative and neutral words.

5. Representation Of Sentiment Results

Using a quick value count, we can see that the sentiment for most of the words is neutral in nature; they are also more positive than negative in sentiment.

As such, we can deduce that the comments of this particular post are neutral-positive skewed.

Removing the neutral words, we can see that the remainder of the words is skewed positive.

Simple Visualizations

Lastly, before ending this post, we can perform some simple visualizations and explore the word frequency of positive and negative words.

Frequency distribution of the 20 most common positive words:

Frequency distribution of the 20 most common negative words:

We can also visualize them via WordClouds.

And the output

Word cloud for positive words

Word cloud for negative words

Lastly, we can visualize the word counts using a bar chart:

Bar chart of the 20 most common positive words by count

Bar chart of the 20 most common negative words by count

There you have it! In this guide, we have learnt how to establish a Reddit instance, create a submission object of our choice, obtain comments from a specific post and apply one of many sentiment analysis models to obtain an overall post's overall sentiment.

We have also learnt how to generate simple visualizations such as word clouds and bar charts of frequent words. I hope you’ve found this mini-project useful.

Cheers!