How to Use the Reddit API in Python | Towards Data Science

Reddit is a huge ecosystem brimming with data that is readily available at our very fingertips. As a data-minded person, I wanted to take advantage of this and perform some analysis using this vast repository of open-source data.
Initially, it turned out that getting to grip with Reddit’s API wasn’t as clear-cut as expected — despite being a straightforward process; it can be a little confusing at first.
So, after figuring everything out, I wrote this article — which I hope will help a few of you to get familiar with using the Reddit API in Python. We will cover:
> Getting Access> Making Requests  - Reading the Data  - Streaming New Posts> Parameters
If you prefer video, I cover everything from this article here too:
First, we need access. Unlike most popular services, the Reddit API was somewhat difficult to figure out initially. There are several steps:
1. Go to App Preferences and click create another app... at the bottom.
2. Fill out the required details, make sure to select script — and click create app.
3. make a note of the personal use script and secret tokens:
4. Request a temporary OAuth token from Reddit. We need our username and password for this:
5. Add headers=headers to every request. The OAuth token will expire after ~2 hours, and a new one will need to be requested.
Now we have everything set up — we can begin making requests. If we want to start pulling Reddit data, our first stop could be to begin reading the top posts in our selected subreddits.
Let’s start with /r/Python. We can retrieve the most popular posts with GET /r/subreddit/hot like so:
We return a lot of data, so let’s filter for something more readable. Each post can be accessed by looping through each item in res.json()['data']['children'].
Each post’s title is stored inside 'title', an item inside the 'data' dictionary — so, we print the title of every post like this:
We can use the same approach to build a dataframe containing basic information about the post. In particular, it may be useful to pull the title, date-time, content, and reactions to a post.
We can modify our code very slightly to return all of the most recent posts instead. Providing us with a stream of posts as they’re made in real-time.
If we look at the API documentation here, we can see that GET /r/subreddit/new it provides the functionality we need. All we need to do us change our get request from /r/python/hot to /r/python/new:
And with that, we have built two methods for scraping data from any subreddit — and we can use essentially the same code for any other method too!
But there is more. There are a few parameters that we can add to our queries. The most relevant for scraping data are:
limit — the maximum number of items to return.
before/after — used to specify that we only want to search for posts that are before or after another.
The limit parameter is pretty self-explanatory. We pass a number to specify that we would like to receive no more than this number of items.
Where applicable, the limit is set to 25 by default. We can pump this up to a maximum value of 100 items.
Next up are the before and after parameters. These both work in the same way — we pass a post fullname, and the API will then retrieve items that are either before or after that post.
Confusingly, after means further back in time, whereas before means more recently in time. So, if we take item t3_k3vndz at 13:21:
before gives us t3_k3vsri at 13:30.
after gives us t3_k3usna at 12:21.
All we need now is to find the fullname of our items. Back in the Reddit API docs, this section explains exactly what we need.
We first need to identify what type of item we are working with. In our example, we have been extracting posts on the /r/Python subreddit. These posts are classified as links — as they do, in fact, link to the post itself.
Rather than guessing what type of item we are returning from the API — we simply check. The type for our returned posts is provided in the items 'kind' key:
Likewise, the ID is provided in our response too. Within the data dictionary, we have an 'id' key. We concatenate these two parts together to create a fullname:
Now that we have a method for building fullnames — we can enhance our data collection to travel further back in time.
To do this, we:
Request the 100 newest posts, storing the response in df.Get the fullname of the oldest post in df.Request the 100 posts that were published after our fullname.Append our new posts to df.Repeat steps 2–4 until satisfied!
Request the 100 newest posts, storing the response in df.
Get the fullname of the oldest post in df.
Request the 100 posts that were published after our fullname.
Append our new posts to df.
Repeat steps 2–4 until satisfied!
With that, we have built a tool for extracting huge amounts of what can be incredibly valuable data from Reddit. The full script for this tool is here:
Once we have set up our API authentication and made a few requests, it becomes much easier to understand how to interact with the API.
The Reddit API documentation is simple, which makes it very easy to understand — I would definitely recommend scrolling through to see what can be done with it.
An alternative is to use the Python PRAW library — a purpose-built library for interacting with the Reddit API. If you’re not too comfortable with requests or would simply appreciate the added layer of abstraction, check it out!
That is all for this introduction to using the Reddit API in Python! I hope it has been useful — if you have any questions or feedback, let me know via Twitter or in the comments below.
Thanks for reading!
Reddit API Docs
🤖 NLP With Transformers Course