Scrape Reddit data using Python and Google BigQuery | by Akhilesh Narapareddy | Towards Data Science

Reddit is one of the oldest social media platforms which is still going strong in terms of its users and content generated every year.Behind that age old user interface, is the treasure trove of information that millions of users are creating on a daily basis in the form of questions and comments.
In this post, we will see how to get data from Reddit website using python and Google Bigquery in a step by step manner.To illustrate this process, I decided to extract the data about cord-cutters, people who cut their cable connection and purchase streaming site subscriptions, as this phenomenon is of an interest to me.
As a first step, let us understand the structure of Reddit website.
The site is divided into various subreddits, with each user choosing which subreddits they would like to subscribe to according to their interest. These include music subreddits where links regarding music may be shared, sports subreddits where people talk in detail about sports or in our case cord-cutter subreddit where people discuss about cable connections or about their new streaming subscriptions.
Upvote and downvote system is the essence of Reddit as it shows an agreement regarding a particular topic among the community members. The more upvotes a post gets, the more prominently it will be displayed on the site. It is important to note that the comments are as important as posts as they often become extended nested discussions.
While web scraping is one among the famous(or infamous!) ways of collecting data from websites, a lot of websites offer APIs to access the public data that they host on their website. This is to avoid unnecessary traffic that scraping bots create, often crashing their websites causing inconvenience for the users. Even Reddit also offers such API which is easy to access.
Following are the things that you will need for this exercise:
Python 3.x : You can download it hereJupyter notebook: We will be using this as our interactive consoleReddit account: You will have to create a Reddit account
Python 3.x : You can download it here
Jupyter notebook: We will be using this as our interactive console
Reddit account: You will have to create a Reddit account
There are a few steps that you need to follow before you get the data for your desired topic.
Create an App:
Create an App:
First step after signing up is to create an app to get Oauth keys for accessing the data. Click here to get started.
Click on the create an app as shown in the snap shot. Then a dialog box appears like the one below.
Enter a name for your app in the dialog box and click on the script as we will be using this for our personal use. Make sure that you enter http://localhost:8080 in redirect URL box. You can refer to praw documentation if you need any clarification. Now click on create app button at the bottom.
Now your application has been created. Store your 14 char personal use script and 27 char secret key somewhere secure. Now you have all the credentials that are required for OAuth2 authentication to connect to Reddit API.
It's time to open Jupyter notebook now!
2.Establish the connection
The packages that we require for this exercise are praw and pandas. PRAW is short for Python Reddit API Wrapper which we will use to make requests to the Reddit API. Make sure that you have both of them installed. First step is to import those packages
import prawimport pandas
Next step after importing the packages is to establish a connection with Reddit API using the credentials that we have created earlier. Client_id will be your 14 char personal use script key and client_secret is your 27 char secret key. Username and password are your Reddit account credentials. Rest of the code will remain the same.
reddit = praw.Reddit(user_agent='Comment Extraction (by /u/USERNAME)',client_id='**********',client_secret="***********",username='********', password='*******')
By running the above snippet of code, we will be establishing the connection and storing this information in a variable named reddit.
3.Getting the data
As we have discussed earlier, we would be focusing on getting the data for 'cordcutter' subreddit.
title, score, url, id, number of comments, date of creation, body text are the fields that are available when it comes to getting the data from Reddit API. But as i will be not considering any time aspect in our analysis, our main focus will be to get just the bodytext(comments) from the subreddit. Refer to praw documentation for different kinds of implementations. Here I have limited the code to the desired output which is just body text for all comments.
To have all the comments including the nested replies, I have to come up with a nested code with 3 parts to it.
Getting the list of comments ids
comm_list = []header_list = []i = 0for submission in reddit.subreddit('cordcutters').hot(limit=2):    submission.comments.replace_more(limit=None)    comment_queue = submission.comments[:]
In this loop, first we are getting each submission information at the start of the loop and then extracting all the comment ids and storing them in list.
Here, .hot(limit) can be any number depending on your requirement. I have set it to 2 here to illustrate the output, but setting it to None will fetch you all the top level submissions in cordcutter subreddit. replace_more(limit=None) will help us in considering the comments that have nested replies.
Output of this loop looks like this:
[Comment(id='ed5ssfg'),  Comment(id='ed64a72'),  Comment(id='edth3nc'),  Comment(id='ed680cg'),  Comment(id='ed699q2'),  Comment(id='ed80ce8'),  Comment(id='edau9st'),  Comment(id='edcx477'),  Comment(id='ee0fp3g'),  Comment(id='ed5qrvh')]
Getting all the nested replies
We are close to getting our desired data. In this part of the code, we will be getting the body of each comment id that we obtained earlier. If the comment has nested replies, it will enter into the next loop and will extract information in a similar fashion.
while comment_queue:    header_list.append(submission.title)    comment = comment_queue.pop(0)    comm_list.append(comment.body)    t = []    t.extend(comment.replies)    while t:        header_list.append(submission.title)        reply = t.pop(0)        comm_list.append(reply.body)
By this point, we downloaded the comments from Reddit and there would required a bit of pre-processing to download it as a csv.
Below is the collated code
comm_list = []header_list = []i = 0for submission in reddit.subreddit('cordcutters').hot(limit=2):    submission.comments.replace_more(limit=None)    comment_queue = submission.comments[:]  # Seed with top-level    while comment_queue:        header_list.append(submission.title)        comment = comment_queue.pop(0)        comm_list.append(comment.body)        t = []        t.extend(comment.replies)        while t:            header_list.append(submission.title)            reply = t.pop(0)            comm_list.append(reply.body)df = pd.DataFrame(header_list)df['comm_list'] = comm_listdf.columns = ['header','comments']df['comments'] = df['comments'].apply(lambda x : x.replace('\n',''))df.to_csv('cordcutter_comments.csv',index = False)
You can find the final version of the code in my github repository.
github.com
Our final output looks like this:
We have our data, but there is one challenge here. Generally, it takes more time to get months of historical data using Reddit API. Thanks to Jason Michael Baumgartner of Pushshift.io(a.k.a /u/Stuck_In_The_Matrix on Reddit), we have years of historical Reddit data cleaned and stored in Bigquery which is the second part of this post.
For those who do not know what Bigquery is,
Google BigQuery is an enterprise data warehouse that solves this problem by enabling super-fast SQL queries using the processing power of Google’s infrastructure.
Best part is querying this data would be free. Google provides first 10GB of storage and first 1 TB of querying memory free as part of free tier and we require less than 1 TB for our task.
Lets look at how to query this information.
First click on this Google BigQuery link to get started. Google will automatically log you in with your Google credentials that are stored in your browser. If this is your first time on BigQuery, there will be a dialog box asking you to create a project.
Hit on create a project button.
Give a name to the organization and click on create project at the top.
Give the name of the project and you can leave the location box as it is for now. Then click on create. Now you will have your project created and a dashboard appears on the screen.
Now after this, click on the link .This will open the reddit datasets under the project that you have created. On the left hand side, you will see datasets updated for each month under the schema name fh-bigquery.
Let’s run the query to get the data for one month from the table.
select subreddit,body,created_utcfrom `fh-bigquery.reddit_comments.2018_08` where subreddit = 'cordcutters'
This will get you all the comments for ‘cordcutter’ subreddit. But make sure that you leave the “Use Legacy SQL” check box in the options unchecked as the above snippet of code is in standard sql. However, you can choose your choice of sql and make changes to the code accordingly.
This is how the result looks like and you can download the result as a csv by clicking “Download as CSV” button.
Here, I have just focused on getting the data as we requried it. If you want to play more with reddit data on bigquery, you can refer this article by Max Woolf which goes into more detail about Reddit data in Bigquery.
In this post, we have seen how to create OAuth2 credentials for connecting to Reddit, making data requests to Reddit API to get most recent data and query historical data in a very fast way through Google Bigquery.
In addition to getting data through an API and Bigquery, you might find it interesting to look at web scraping using Selenium and python. Following is an article about that by a fellow classmate(Atindra Bandi) at UT Austin.
towardsdatascience.com
That’s all folks! Stay tuned to get an update on a series of articles on recommendation systems, statistics for data science and data visualizations from me in the coming weeks.