Scraping Reddit with PRAW. Recently I was trying to get started on... | by McGuckian Shane | Towards Data Science

Recently I was trying to get started on a project that would use Natural Language Processing to classify which subreddit a given post came from. For instance, this model should be able to predict whether or not a post came from the r/Python subreddit or the r/Rlanguage subreddit. The first step in this process was to collect a number of posts from each subreddit. In the past I’ve used the BeautifulSoup and requests python libraries to get this done. Some of my colleagues had mentioned that API wrappers can be exceedingly useful and simplify a large part of the process. This is how I stumbled upon The Python Reddit API Wrapper (PRAW).
One of the most helpful articles I found was Felippe Rodrigues’ “How to Scrape Reddit with Python.” He does a great job of walking through the basics and getting set up. Definitely check it out if you’re interested in doing something similar. I used this article as a jumping off point for my own project. Below I’ll walk you through how to get setup using PRAW and how to scrape post titles, content, and other metadata. You can find the complete code on my personal Github.
The first step required to start using PRAW is to create an application through reddit. If you go to this link you should find a button near the bottom labeled “create app” or “create another app.” You’ll have to give your script a name and fill out a description. Once that’s done, make sure to select the “script” option and then make sure to put the following into the redirect uri box: http://localhost:8080. This is suggested by the PRAW docs, but is apparently necessary because Reddit requires a redirect uri even if our application doesn’t use it. If successful, you should be provided with two strings of seemingly random characters. You can find the first, your “personal use script,” near the top left corner directly under where it says “personal use script.” This one should be 14 characters long. The next thing you’ll need is the “secret”. While it may sound like something out of a fantasy novel it should be represented as a 27 character string listed just below the “personal use script”. With these two things we can finally begin our journey of scraping posts from Reddit!
Now you need to open your favorite editor or IDE. I used PyCharm for this project but Atom or Sublime would work just fine. The first thing we need to create is a praw.ini file. This will allow us to utilize the PRAW Reddit instance in the future, and it means that we can leave our personal use script and secret out of our primary script if you decide to publish it.
You can refer to the docs regarding praw.ini files, but do make sure that file is in fact called “praw.ini” and that it’s located in the same directory as the one your scraping script will be in. It is possible to define multiple credentials in this file, but today we’ll only be using one so we’ll stick with the [DEFAULT] site.
[DEFAULT]; this is your 14 character personal use scriptclient_id=51IfSlyxtyOUy3; this is your 27 character secretclient_secret=NToU1zG5d0LJq9fo3ryUtOigM5h; this is the name you gave your applicationuser_agent=subreddit scraper; this is username for the reddit account the app was created withusername=fake_username; password for the accountpassword=fake_password
Make sure to fill in each of these fields with your information. The fields above were filled in using randomly generated and fake values. It will not work if you don’t replace these fields with your own information.
Now we can begin writing the actual scraping script. The first step is to import the necessary libraries and instantiate the Reddit instance using the credentials we defined in the praw.ini file.
from os.path import isfileimport prawimport pandas as pdfrom time import sleep# Get credentials from DEFAULT instance in praw.inireddit = praw.Reddit()
I decided to create to a class that would allow me to specify the particular subreddit I was interested in, the method of sorting, number of posts, and whether or not the results should be written to a file. Here’s what the __init__ method looks like:
class SubredditScraper:    def __init__(self, sub, sort='new', lim=900, mode='w'):        self.sub = sub        self.sort = sort        self.lim = lim        self.mode = mode        print(            f'SubredditScraper instance created with values '            f'sub = {sub}, sort = {sort}, lim = {lim}, mode = {mode}')
I included a few print statements to get a feel for the script’s progress since it can take a little while to run when working with hundreds of posts. Next, we’ll set the sorting method for our subreddit instance.
def set_sort(self):    if self.sort == 'new':        return self.sort, reddit.subreddit(self.sub).new(limit=self.lim)    elif self.sort == 'top':        return self.sort, reddit.subreddit(self.sub).top(limit=self.lim)    elif self.sort == 'hot':        return self.sort, reddit.subreddit(self.sub).hot(limit=self.lim)    else:        self.sort = 'hot'        print('Sort method was not recognized, defaulting to hot.')        return self.sort, reddit.subreddit(self.sub).hot(limit=self.lim)
This method will set the subreddit and sorting parameters of the Reddit instance to the values we specify when instantiating our class. This will return a tuple that we’ll unpack in the next method. If the sorting method is not new, top, or hot it will default to hot.
Finally, we can begin collecting posts and other information from the specified subreddit:
def get_posts(self):    """Get unique posts from a specified subreddit."""    sub_dict = {        'selftext': [], 'title': [], 'id': [], 'sorted_by': [],        'num_comments': [], 'score': [], 'ups': [], 'downs': []}    csv = f'{self.sub}_posts.csv'    # Attempt to specify a sorting method.    sort, subreddit = self.set_sort()    # Set csv_loaded to True if csv exists since you can't     # evaluate the truth value of a DataFrame.    df, csv_loaded = (pd.read_csv(csv), 1) if isfile(csv) else ('', 0)    print(f'csv = {csv}')    print(f'After set_sort(), sort = {sort} and sub = {self.sub}')    print(f'csv_loaded = {csv_loaded}')    print(f'Collecting information from r/{self.sub}.')
Here we’re creating a placeholder dictionary with each of attributes we’ll be collecting from each of the posts for this subreddit. Then we’re setting up the .csv file for future use, and we’re unpacking the tuple returned by the set_sort method. The last piece of setup is to see if we’ve previously collected posts for this subreddit. If so, we’ll load that .csv file into a data frame and set the boolean variable csv_loaded to 1. If not, df will be an empty string and csv_loaded will be set to 0.
This brings us to the real meat of the scraping. We’ll use a for loop to look at each post and collect the attributes we’re interested in.
for post in subreddit:    # Check if post.id is in df and set to True if df is empty.    # This way new posts are still added to dictionary when df = ''    unique_id = post.id not in tuple(df.id) if csv_loaded else True    # Save any unique posts to sub_dict.    if unique_id:        sub_dict['selftext'].append(post.selftext)        sub_dict['title'].append(post.title)        sub_dict['id'].append(post.id)        sub_dict['sorted_by'].append(sort)        sub_dict['num_comments'].append(post.num_comments)        sub_dict['score'].append(post.score)        sub_dict['ups'].append(post.ups)        sub_dict['downs'].append(post.downs)    sleep(0.1)
If you plan to use this script multiple times to collect a large number of posts we’ll need to check whether or not each post is unique or if we’ve previously added it to our .csv file. This is where our boolean variable csv_loaded will come in handy. We’ll check if the id attribute of our post is in the “id” column of our data frame (df). If this is the first time we’ve collected posts for this subreddit, unique_id will be set to True for every post. Then we’ll append each of the post’s attributes that we’re interested in to our placeholder dictionary. Finally, and possibly most importantly we’ll sleep for a tenth of a second after every post. This serves as a kind of artificial rate limiter. If we don’t do this we’ll eventually hit the API’s request limit (1000). There are several ways to get around this limit, one of which is to request a refresh token, but this should work fine for the time being.
Next we’ll save our results to a .csv or modify the existing one depending on whether or not we’ve scraped this subreddit before:
new_df = pd.DataFrame(sub_dict)# Add new_df to df if df exists then save it to a csv.if 'DataFrame' in str(type(df)) and self.mode == 'w':    pd.concat([df, new_df], axis=0, sort=0).to_csv(csv, index=False)    print(        f'{len(new_df)} new posts collected and added to {csv}')elif self.mode == 'w':    new_df.to_csv(csv, index=False)    print(f'{len(new_df)} posts collected and saved to {csv}')else:    print(        f'{len(new_df)} posts were collected but they were not '        f'added to {csv} because mode was set to "{self.mode}"')
Here we’re using pandas concat method in order to add our new results to the existing .csv we loaded into the df data frame if it exists and mode was set to “w” when our class was instantiated. If there wasn’t an existing .csv file for this subreddit we’ll write our results to .csv, and if the mode wasn’t set to “w” we’ll print out the number of posts we found instead of writing them to a file.
That’s all that we need to round out our SubredditScraper class. The last thing left to do is instantiate our new class.
if __name__ == '__main__':    SubredditScraper(        'python',         lim=997,         mode='w',         sort='new').get_posts()
If you’re not familiar with the use of if __name__ == '__main__': I suggest checking out Corey Schafer’s video on this topic here. With that, you should be able to run this script, collect almost 1000 posts from the python subreddit, and save those posts and some of their metadata to a .csv file. I hope this was helpful! Don’t forget you can find the complete code for this project here.