Reddit data analytics trilogy #1 — Data scraping with PRAW
By: Huifang Yeo
Tags: Reddit, Real Time Analytics, Nlp Training, Data Visualization, Use Cases

Reddit data analytics trilogy #1 — Data scraping with PRAW

Fun scraping Reddit data

I was inspired by an article about creating a Data Table using Data from Reddit and I was convinced that I could do much more with atoti. Hence I started digging into the Reddit data and it piqued my interest so much, it became a trilogy! In this trilogy, I’m going to take you through:

The article mentioned above was published quite some time ago, and Dash would probably have evolved way beyond. Nonetheless, my end game is to achieve the below dashboard with atoti:

Check out how I could drill-down on different dimensions in the pivot table and interact with different data visualizations.

Let’s start with part 1 of my trilogy — data scraping Reddit! If you haven’t heard of Reddit before, do take a look! It has tremendous amounts of information on pretty much everything, classified under communities known as subreddits.

Taken from Subreddit r/dataisbeautiful — something I could totally relate to

Being community driven, Reddit is a treasure trove of data that gives the trends and the opinions of the community towards various topics. Let’s take a look at how you can get your hands on these data.

PRAW — Python Reddit API Wrapper

PRAW is a Python package that I used to access Reddit’s API to scrape the subreddits that I’m interested in. For your information, you need to have a Reddit account in order to get an API key that is necessary for connection to Reddit.

Click here to proceed to create an app by clicking on the button shown on the left.

Select “script” in order to obtain “refresh_tokens” and also use “http://localhost:8080” for the redirect uri as mentioned in the PRAW documentation.

Click on the “create app” to get the API information needed for Reddit connectivity.

PRAW — Authentication

You can store the sensitive information highlighted in the screenshot above separately, and invoke properly when needed. To keep things simple, I’m showing how to apply these values to connect to Reddit. Pluck the 3 values into the code segment below:

Accessing Subreddits

This is where the fun begins! Using the authorized Reddit instance, I can obtain a subreddit instance by passing the name of the subreddit as follows:

You can also combine multiple subreddits as follows:

Have a look at the Metrics for Reddit to know what Subreddits are available and which are the popular ones.

Accessing Submission instance from Subreddit instance

Reddit front is a listing class that represents the front page. Below is a summary of the front methods and what type of submissions they return:

best — best items

comments — most recent comments

controversial — controversial submissions

gilded — gilded items

hot — hot items

new — new items

random_rising — random rising submissions

rising — rising submissions

top — top submissions

To access the attributes of the submission, iterate through the list returned by the front method.

In my case, I appended the attributes of the submission to a list so that I can convert it into Pandas dataframe later on:

Notice that I used the front method “new” to retrieve the latest 100 submission (set using the limit parameter) from the subreddit Wallstreetbets.

Voila! Now I have the data from Reddit and I’m ready to try out Natural Language Processing (NLP) with it! Check out part 2 of my trilogy — NLP with spaCy!