How to Access Twitter’s API using Tweepy | by Paulynn Yu | Towards Data Science

Tweet datasets are an extremely desirable corpus for aspiring (and practicing) data scientists to analyze and perform models on. Tweets are by nature short-form and contain diverse and relevant topics, making it an excellent dataset for sentiment analysis. It is also a great dataset to analyze user engagement since tweet timestamps are available.
While there are many existing Twitter datasets out there, they will be predefined for you. I am a believer of first defining the problem you’re interested in, and then figuring out a way to get that data. This is where it is beneficial to access Twitter API — you get the type, volume and ‘newness’ only an API can provide.
If it’s any testament, I was able to obtain 70,000 user accounts and their 10 million post which takes close to a full day. It’s slightly longer than proofing your bread dough for homemade sourdough, but really not that much longer. I recommend it more highly than trying to make any bread from scratch (at least once) which believe me, says a lot.
The Twitter API exposes dozens of HTTP endpoints that can be used to retrieve, create and delete tweets, retweets and likes. It provides direct access to rich and real-time tweet data, but requires having to deal with a lot of low level details (and very not fun debugging).
Tweepy is an open source package that allows you to bypass a lot of those low level details. Twitter’s developer website has great documentation I recommend exploring to get example responses and see the type of data you are able to access. Tweepy’s documentation will furthermore provide code snippets and some basic documentation for the Tweepy module.
Twitter API uses OAuth, which is an open authorization protocol to authenticate requests. You will need to create and configure your authentication credentials to access Twitter API. As promised, this is a step-by-step guide so follow along!
Step 0: Open a Twitter account. If you already have a Twitter account, skip this step
Step 1: Apply for a developer account
Go to their developer site and go to apply for access and select “Apply for a developer account”. You will be prompted to log in to your Twitter account.
You will then be navigated to a page like this:
Select your choice path and fill the details in the next page and fill in some personal details. When you get to intended use, there are a couple of fields with minimum character limit (the one time they have a minimum 😏).
This is what I filled up. Be honest, but don’t worry too much about the details and especially not about brevity. In my experience, they have been instantaneous in approvals.
There will be further easy questions and steps (be patient!). Once you’ve gone through those steps and accept developer agreement, you now have a developer account!
Step 2: Create an Application
You might have to wait a hot minute for your developer account to be approved — but once it is, you can start creating your application.
Go to your profile tab and select Apps. Create an app and fill in the details. That should take another hot minute.
Once you create you app, go to the next step.
Step 3: Get your authentication details
Go to your apps page where you will see the app you created. Click on details.
Once you’re there, click on keys and tokens to get the relevant keys. You will might need to generate your access token and access token secret.
You also have the capability to regenerate the key, in case you are needing to write a step-by-step blog post and share your old key (I did).
Once you have that, move to to the next step!
Replace the CONSUMER_KEY, CONSUMER_SECRET, ACCESS_TOKEN, ACCESS_TOKEN_SECRET in the code below with your own credentials.
Run the code below to verify your authentication. I hope it works!
import tweepy# Authenticate to Twitterauth = tweepy.OAuthHandler("CONSUMER_KEY", "CONSUMER_SECRET")auth.set_access_token("ACCESS_TOKEN","ACCESS_TOKEN_SECRET")api = tweepy.API(auth)# test authenticationtry:    api.verify_credentials()    print("Authentication OK")except:    print("Error during authentication")
Tweepy has a list for methods that will easily help you access Twitter’s end points. There is a method for user timeline, tweets, searches, trends, users and more. Do read documentation for full list, but let me talk about these two methods I used.
follower_ids
This method allows you to get most recent following of a particular user (use screen_name as parameter). Results are given in groups of 5,000 user IDs and you can use the cursor to navigate through the ‘pages’.
api.followers_ids(screen_name=screen_name)
For my purposes, I used this as a strategy to get a list of IDs as a parameter to feed into user_timeline method
user_timeline
The overall rate limit to this method is 100,000 calls during any single 24-hour period. That will translate to 100,000 users and their timeline posts (up to 200 most recent posts).
timeline = api.user_timeline(user_id=user_id, count=200)
The below is a snippet of the JSON that it will return. Do look through the attributes that are useful for your purposes.
For me, the attributes I collected were [‘created_at’, ‘text’, ‘source’, ‘in_reply_to_screen_name’, ‘retweet_count’, ‘favorite_count’, ‘favorited’, ‘retweeted’, ‘is_quote_status’, ‘retweeted_status’, ‘hashtags’, ‘symbols’, ‘user_mentions’]
search
I did not use this method, but I believe this is useful for most of you seeking Twitter data to get conversations on a particular topic. This method returns a collection of relevant Tweets matching a specified query for all public tweets.
The example below returns the 5 most recent tweets about hot pockets Snowden.
api.search(q="hot pockets snowden", lang="en", rpp=5)
The results you need to parse are:
This function will help you get the list of follower IDs for a given Twitter screen name. Each page is has 5000 IDs so the cursor just helps you ‘flip’ through the pages if the particular user has more than 5000 followers. My plan was to call the user timeline for each of this IDs.
# define screen_namedef get_ids(screen_name):    '''    :argument: screen_name of user    :returns: a list_id of the given user's followers    '''    # get first list    first_list = api.followers_ids(screen_name=screen_name)    id_list = first_list['ids']    cursor = first_list['next_cursor']    while cursor != 0 :        user_ids = api.followers_ids(screen_name=screen_name,  cursor=cursor)        id_list.extend(user_ids[0]['ids'])        cursor = user_ids[0]['next_cursor']    return id_list
The below function is a bit of a handful. But high level, it gets tweets based on attributes I specified. There were some simple manipulations (replace function) I had to perform to help with next line (\n) formatting from the API calls. This is entirely for my operation as my final was a csv file which I move to a postgresql database.
If you are capturing this in MongoDB, this might not be needed.
def get_tweets(user_id, timeline) :    '''    :param user_id: the list_id as recognized by twitter    :param timeline: the dictionary that is pulled from user_timeline twitter API    :return: all the posts in dictionary format of a specified list_id    '''    # attribute lists    post_attrs = ['created_at', 'text', 'source', 'in_reply_to_screen_name', 'retweet_count', 'favorite_count',                  'favorited', 'retweeted', 'is_quote_status', 'retweeted_status']    entities_attrs = ['hashtags', 'symbols', 'user_mentions']     # creating empty dictionary, and specifying user id    post_dict = defaultdict(list)    for tweets in timeline :       post_dict['user_id'] = user_id           for post in list(post_attrs) :              if post == 'text' :                try :                    t = tweets[post]                    t = t.replace('\n','')                    t = t.replace('\r','')                    post_dict[post].append(t)                except :                    post_dict[post].append(np.nan)            else :                try :                    t = tweets[post]                    post_dict[post].append(t)                except :                    post_dict[post].append(np.nan)      # looping through other post attributes        for entity in entities_attrs :            try :                attr_name = 'len_' + entity                post_dict[entity].append(tweets['entities'][entity])                post_dict[attr_name].append(len(tweets['entities'][entity]))            except :                post_dict[entity].append(np.nan)    return post_dict
auth = tweepy.OAuthHandler("CONSUMER_KEY", "CONSUMER_SECRET")auth.set_access_token("ACCESS_TOKEN","ACCESS_TOKEN_SECRET")api = tweepy.API(auth, wait_on_rate_limit=True, wait_on_rate_limit_notify=True, parser=tweepy.parsers.JSONParser())
In your API method, do specify wait_on_rate_limit and wait_on_rate_limit_notify which will wait once you’ve reached your rate limit and prints out a message.I found that calling the API from an AWS server (or whatever cloud server) was quicker than using my local machine. This might be purely my own heuristics from observing how quickly printed users IDs were populating as I queried the API. You can try both, especially if speed is important to you.This is probably the biggest tip I can give you and was game-changing in my own data collection. Despite Twitter stating a request limit of 900 per 15 minutes, they will become much much slower (1 call per 7 seconds vs 1 per second) if you’re requesting continually in one batch. By splitting my batches to 5,000 at a time (rather than 20,000 in my first iteration), I was able to get user historical tweets much quicker. What this means is to run your user ID iteration 5000 users at a time then take a break, make another API auth call and call another 5,000 users. Do feel free to test in even smaller batches.
In your API method, do specify wait_on_rate_limit and wait_on_rate_limit_notify which will wait once you’ve reached your rate limit and prints out a message.
I found that calling the API from an AWS server (or whatever cloud server) was quicker than using my local machine. This might be purely my own heuristics from observing how quickly printed users IDs were populating as I queried the API. You can try both, especially if speed is important to you.
This is probably the biggest tip I can give you and was game-changing in my own data collection. Despite Twitter stating a request limit of 900 per 15 minutes, they will become much much slower (1 call per 7 seconds vs 1 per second) if you’re requesting continually in one batch. By splitting my batches to 5,000 at a time (rather than 20,000 in my first iteration), I was able to get user historical tweets much quicker. What this means is to run your user ID iteration 5000 users at a time then take a break, make another API auth call and call another 5,000 users. Do feel free to test in even smaller batches.
That’s all for now — please let me know if you have any questions. Happy requesting live fresh data!