Ultimate Beginners Guide to Collecting Text for Natural Language Processing (NLP) with Python — Twitter, Reddit, Genius and More | by Eric Kleppen | Towards Data Science

I’ve been fascinated by Natural Language Processing (NLP) since I got into data science a little over a year ago. Thanks to advancements in transfer learning, there has been some explosive progress in the field, and NLP products like Alexa and Siri have become household names. Since my background is technical writing and rhetorical theory, I was immediately drawn to projects involving text like sentiment analysis and topic extraction because I wanted to develop an understanding of how machine learning can provide insight into written language. One of my first data science projects was using Google’s Universal Sentence Encoder to produce wine recommendations.
I always wanted a guide like this one, breaking down how to extract data from popular social media platforms. With increasing accessibility to powerful pre-trained language models like BERT and ELMo, it is important to understand where to find and extract data. Luckily, social media is an abundant resource for collecting NLP data sets, and is easily accessible with a few lines of Python. At the end of the article, I also include a list of popular Kaggle NLP datasets, and link to Google Dataset Search, the new search engine.
This article teaches you how to extract data from Twitter, Reddit, and Genius. I assume you already know some Python libraries Pandas and SQLite.
Before getting into the code, it is important to stress the value of an API Key. If you’re new to managing API keys, make sure to save them into a config.py file instead of hard-coding them in your app. Make sure not to include them in any code share online. API keys can be very valuable, and sometimes very expensive and must be protected. If you’re worried your key has been leaked, most providers allow you to regenerate them.
Add the config file to your gitignore file to prevent it from being pushed to your repo too!
Twitter provides a plethora of data that is easy to access through their API. With the Tweepy Python library, easily pull a constant stream of tweets based on the desired topics. Twitter is great for mining trends and sentiment,
For this tutorial, you will need to register an app with Twitter to get API Keys. Check out the official Twitter documentation if you’re not familiar with their developer portal!
Use pip to install Tweepy and unidecode.
pip install tweepypip install unidecode
Save the following keys to a config file:
Connecting Tweepy to Twitter uses OAuth1. If you’re brand new to API authentication, check out the official Tweepy Authentication Tutorial.
To save the data from the incoming stream, I find it easiest to save it to an SQLite database. If you’re not familiar with SQL tables or need a refresher, check this free site for examples or check out my SQL tutorial.
The function unidecode() takes Unicode data and tries to represent it in ASCII characters.
#import dependenciesimport tweepyfrom tweepy import OAuthHandlerfrom tweepy.streaming import StreamListenerimport jsonfrom unidecode import unidecodeimport timeimport datetime#import the API keys from the config file.from config import con_key, con_sec, a_token, a_secret sqlite3conn = sqlite3.connect("twitterStream.sqlite")c = conn.cursor()
I need to create the table to store the wine data. I use SQLite because it is lightweight and server-less. Plus I like keeping all the data in once place!
def create_table():    c.execute("CREATE TABLE IF NOT EXISTS Tweets(timestamp REAL, tweet TEXT)")    conn.commit()create_table()
Notice I use IF NOT EXISTS to make sure the table doesn’t already exist in the database. Remember to commit the transaction using the conn.commit() call.
Here is some boilerplate code to pull the tweet and a timestamp from the streamed twitter data and insert it into the database.
class Listener(StreamListener):        def on_data(self, data):        try:            data = json.loads(data)            tweet = unidecode(data['text'])            time_ms = data['timestamp_ms']            #print(tweet, time_ms)            c.execute("INSERT INTO Tweets (timestamp, tweet) VALUES (?, ?)", (time_ms, tweet))                        conn.commit()            time.sleep(2)        except KeyError as e:                            print(str(e))        return(True)            def on_error(self, status_code):        if status_code == 420:            #returning False in on_error disconnects the stream            return Falsewhile True:       try:            auth = OAuthHandler(con_key, con_sec)            auth.set_access_token(a_token, a_secret)            twitterStream = tweepy.Stream(auth, Listener())            twitterStream.filter(track=['DataScience'])       except Exception as e:            print(str(e))            time.sleep(4)
Notice I slow the stream using time.sleep().Notice the code is wrapped in a try/except to prevent potential hiccups from disrupting the stream. Additionally, the documentation recommends using an on_error() function to act as a circuit-breaker if the app is making too many requests.Notice I wrap the stream object in a while condition. That way, it stops if it hits the 420 error.Notice the twitterstream.filter uses track to find keywords in tweets. If you want to follow a specific user’s tweets, use .filter(follow=[“”]).
Extract the data from the SQLite database
sql = '''select tweet from Tweets                 where tweet not like 'RT %'                order by timestamp desc'''tweet_df = pd.read_sql(sql, conn)tweet_df
Like Twitter, the social network Reddit contains a jaw dropping amount of information that is easy to scrape. It is a social network that works like an internet forum allowing users to post about whatever topic they want. Users form communities called subreddits, and they up-vote or down-vote posts in their communities to decide what gets viewed first and what sinks to the bottom.
I’ll explain how to get a Reddit API key and how to extract data from Reddit using the PRAW library. Although Reddit has an API, the Python Reddit API Wrapper, or PRAW for short, offers a simplified experience. PRAW supports Python 3.5+
A user account to Reddit is required to use the API. It is completely free and only requires an email address!
https://www.reddit.com
If there is a way to get here using the new Reddit UI, leave me a comment! If it is your first time, follow these steps to get an API key after signing into Reddit. If you already have a key, use this link to go to your apps page.
Click the User Account droplist. User options display.
Click Visit Old Reddit from the user options. The page will change and the URL will become https://old.reddit.com/
Click the preferences link next to the logout button.Click the apps tab on the PREFERENCES screen.Click the are you a developer? create am app... button.
Enter a name.Select the type of app.Enter a Description.Use http://localhost:8080 as the redirect uri.Click create app after populating the fields.
The API information required to connect will display. I’ll walk through connecting to the API using PRAW when I get into code.
Congratulations on getting set up to scrape Reddit data!
The recommended way to install PRAW is to use pip. The install the following packages to create the dashboard.
pip install praw
Start by importing the libraries and the config file:
import prawimport pandas as pd from config import cid, csec, ua
Create a read-only Reddit instance. That means I don’t need to enter Reddit credentials used to post responses or create new threads; the connection only reads data.
PRAW uses OAuth authentication to connect to the Reddit API.
#create a reddit connectionreddit = praw.Reddit(client_id= cid,                     client_secret= csec,                     user_agent= ua)
Here is a list of examples I think would be fun to explore:
news, datascience, learnmachinelearning, gaming, funny, politics
Use the Subreddit class in PRAW to retrieve the data from the desired subreddit. It is possible to order the data based on the follow Reddit options:
hot — order by the posts getting the most traffic
new — order by the newest posts in the thread
top — order by the most up-voted posts
rising — order by the posts gaining popularity
If you want to include multiple subreddits, use a + symbol:
#single subreddit new 5subreddit = reddit.subreddit('news').new(limit = 5)#multiple subreddits top 5subreddit = reddit.subreddit('news' + 'datascience').top(limit = 5)
This returns an object that holds the data in an attribute. The attribute is like a key in a dictionary.
The data is linked to an attributed owned by the object. If the attribute is the Key, the data is the Value. The attributes are dynamically generated, so it is best to check what is available using Python’s built-in vars() function.
Use this boilerplate code to see all the attributes owned by object representing the reddit post. It is a LONG list!
subreddit = reddit.subreddit('news').new(limit = 1)for post in subreddit:    pprint.pprint(vars(post))
Notice in the list the attributes of interest:
title — Returns post title.score — Returns number of up-votes or down-votes.num_comments — Returns the number of comments on the thread.selftext — Returns the body of the post.created — Returns a timestamp for the post.pinned — Indicates whether the thread was pinned.total_awards_received — Returns number of awards received by the post.
Now that the attributes have been identified, load them data into a pandas DataFrame or save them to an SQLite database like in the Twitter example. In this example, I’ll save it to a pandas DataFrame.
#list for df conversionposts = []#return 100 new posts from wallstreetbetsnew_bets = reddit.subreddit('wallstreetbets').new(limit=100)#return the important attributesfor post in new_bets:    posts.append([post.title, post.score, post.num_comments, post.selftext, post.created, post.pinned, post.total_awards_received])#create a dataframeposts = pd.DataFrame(posts,columns=['title', 'score', 'comments', 'post', 'created', 'pinned', 'total awards'])#return top 3 df rowsposts.head(3)
I have always been a fan of music, particularly heavy metal. In heavy metal, the lyrics can sometimes be quite difficult to understand, so I go to Genius to decipher them. The website Genius.com is a platform for annotating lyrics, and collecting trivia about music, albums and artists. Genius allows users to register an API Client.
https://genius.com/api-clients
Either Sign up or Sign in. Click the Developers link.Click Create an API Client.Enter an App Name.Enter a website URL if you have one, otherwise http://127.0.0.1 will work.Click Save. The API Client will display.Click Generate Access Token to generate an access token.
Surprisingly, due to legal reasons, the Genius API does not provide a way to download song lyrics. It is possible to search lyrics, but not download them. Lucky for everyone, Medium author Ben Wallace has provided us with a convenient wrapper for scraping the lyrics. Find his original code on GitHub too:
github.com
I have modified his wrapper to make it easier to download an artist’s complete works rather than code the albums I want to include, and I added an Artist column to store the artist name.
The wrapper uses the API to get the URLs linking to the lyrics. From there, BeautifulSoup is used to parse the HTML for each URL. The process results in a dataframe that contains the Title, URL, Artist, Album and Lyrics:
The wrapper is a class named GeniusArtistDataCollect(). Use it to connect to the API and retrieve song lyrics for a specified artist. In the example, I use one of my favorite metal bands, The Black Dahlia Murder.
To use GeniusArtistDataCollect(), instantiate it, passing in the Client Access Token and the Artist name.
g = GeniusArtistDataCollect(token, 'The Black Dahlia Murder')
Call get_artists_songs() from the GeniusArtistDataCollect object. This will return as a pandas DataFrame.
songs_df = g.get_artist_songs()
Here is the modified wrapper I use in the example:
import osimport reimport requestsimport pandas as pdimport urllib.requestfrom bs4 import BeautifulSoupfrom config import tokenclass GeniusArtistDataCollect:    """A wrapper class that is able to retrieve, clean, and organize all the album songs of a given artist    Uses the Genius API and webscraping techniques to get the data."""def __init__(self, client_access_token, artist_name):        """        Instantiate a GeniusArtistDataCollect object        :param client_access_token: str - Token to access the Genius API. Create one at https://genius.com/developers        :param artist_name: str - The name of the artist of interest        THIS HAS BEEN REMOVED :param albums: list - A list of all the artist's albums to be collected        """self.client_access_token = client_access_tokenself.artist_name = artist_name#self.albums = albumsself.base_url = 'https://api.genius.com/'self.headers = {'Authorization': 'Bearer ' + self.client_access_token}self.artist_songs = Nonedef search(self, query):        """Makes a search request in the Genius API based on the query parameter. Returns a JSON response."""request_url = self.base_url + 'search'        data = {'q': query}        response = requests.get(request_url, data=data, headers=self.headers).json()return responsedef get_artist_songs(self):        """Gets the songs of self.artist_name and places in a pandas.DataFrame"""# Search for the artist and get their id        search_artist = self.search(self.artist_name)        artist_id = str(search_artist['response']['hits'][0]['result']['primary_artist']['id'])print("ID: " + artist_id)# Initialize DataFrame        df = pd.DataFrame(columns=['Title', 'URL'])# Iterate through all the pages of the artist's songs        more_pages = True        page = 1        i = 0        while more_pages:print("page: " + str(page))# Make a request to get the songs of an artist on a given page            request_url = self.base_url + 'artists/' + artist_id + '/songs' + '?per_page=50&page=' + str(page)            response = requests.get(request_url, headers=self.headers).json()print(response)# For each song which the given artist is the primary_artist of the song, add the song title and            # Genius URL to the DataFrame            for song in response['response']['songs']:if str(song['primary_artist']['id']) == artist_id:title = song['title']                    url = song['url']df.loc[i] = [title, url]                    i += 1page += 1if response['response']['next_page'] is None:                more_pages = False# Get the HTML, Album Name, and Song Lyrics from helper methods in the class        df['Artist'] = self.artist_name        df['html'] = df['URL'].apply(self.get_song_html)        df['Album'] = df['html'].apply(self.get_album_from_html)        #df['InAnAlbum'] = df['Album'].apply(lambda a: self.is_track_in_an_album(a, self.albums))        #df = df[df['InAnAlbum'] == True]        df['Lyrics'] = df.apply(lambda row: self.get_lyrics(row.html), axis=1)del df['html']self.artist_songs = dfreturn self.artist_songsdef get_song_html(self, url):        """Scrapes the entire HTML of the url parameter"""request = urllib.request.Request(url)        request.add_header("Authorization", "Bearer " + self.client_access_token)        request.add_header("User-Agent",                           "curl/7.9.8 (i686-pc-linux-gnu) libcurl 7.9.8 (OpenSSL 0.9.6b) (ipv6 enabled)")        page = urllib.request.urlopen(request)        html = BeautifulSoup(page, "html")print("Scraped: " + url)        return htmldef get_lyrics(self, html):        """Scrapes the html parameter to get the song lyrics on a Genius page in one, large String object"""lyrics = html.find("div", class_="lyrics")all_words = ''# Clean lyrics        for line in lyrics.get_text():            all_words += line# Remove identifiers like chorus, verse, etc        all_words = re.sub(r'[\(\[].*?[\)\]]', '', all_words)# remove empty lines, extra spaces, and special characters        all_words = os.linesep.join([s for s in all_words.splitlines() if s])        all_words = all_words.replace('\r', '')        all_words = all_words.replace('\n', ' ')        all_words = all_words.replace('  ', ' ')return all_wordsdef get_album_from_html(self, html):        """Scrapes the html parameter to get the album name of the song on a Genius page"""parse = html.findAll("span")album = ''for i in range(len(parse)):            if parse[i].text == 'Album':                i += 1                album = parse[i].text.strip()                breakreturn album
The Genius Lyrics example uses Beautiful Soup to scrape the lyrics from the website. Web scraping is a useful technique that makes it easy to collect a variety of data. I walk through an additional web scraping example in a previous article. Check it out if you want more practice!
towardsdatascience.com
Although I haven’t yet used his methods, Medium writer Will Koehrsen has a walk through for scraping and parsing Wikipedia. Check out his work!
towardsdatascience.com
Although I think it is fun to collect and create my own datasets, Kaggle and Google’s Dataset Search offer convenient ways to find structured and labeled data. Kaggle is a popular competitive Data Science platform. Below is a list of popular datasets used for NLP projects.
https://www.kaggle.com/datasetshttps://datasetsearch.research.google.com/
Determine whether news story is from the Onion or not:
https://www.kaggle.com/chrisfilo/onion-or-not
Youtube rankings and descriptions:
https://www.kaggle.com/datasnaek/youtube-new
Netflix shows and descriptions:
https://www.kaggle.com/shivamb/netflix-shows
Wine Reviews. I use this in several articles and projects:
https://www.kaggle.com/zynicide/wine-reviews
A collection of amazon food reviews:
https://www.kaggle.com/snap/amazon-fine-food-reviews
News headlines for fake news detection:
https://www.kaggle.com/rmisra/news-headlines-dataset-for-sarcasm-detection
Corpus for Entity Recognition tasks.
https://www.kaggle.com/abhinavwalia95/entity-annotated-corpus
Twitter Airline Sentiment tweets for sentiment analysis
https://www.kaggle.com/crowdflower/twitter-airline-sentiment
Yelp review dataset
https://www.kaggle.com/yelp-dataset/yelp-dataset
As NLP becomes more mainstream, it is important to understand how to easily collect rich, text-based datasets. Getting into Natural Language Processing can be tough, so I wanted to share a guide that simplifies ways to collect text data. With a few lines of Python, the astounding amount of data available on Reddit, Twitter, and Genius everyone’s fingertips! Thanks for reading, and check out my other NLP related articles if you want to use the data you know how to collect:
If you enjoyed this, follow me on Medium for more
Get FULL ACCESS and help support my content by subscribing
Let’s connect on LinkedIn
Analyze Data using Python? Check out my website
— Eric Kleppen