Collect Data from Twitter: A Step-by-Step Implementation Using Tweepy | by Zoumana Keita | Towards Data Science

Getting data comes as the second step in any data science/machine learning project lifecycle, right after framing the problem you want to solve, which would make this step be the backbone of the rest of the phases. Also, social media are great places to collect data, especially for competitor analysis, topic research, sentiment analysis, etc. This article aims to perform a step-by-step implementation on how to get credentials and the implementation on a simple use case.
Twitter has 313,000,000 active users (Statista, 2017), which means that this method of data collection could reduce barriers to research participation based on the geographical location of researchers and research resources. It can also maximize resources, including time, effort, and convenience: Sage Journal
Before performing any kind of analysis, the first action to perform is to get your Twitter authentication credentials, as described below.
Get access to authentication credentials
Sign up for developer account
Sign up for developer account
Here is the link that gives access to the following page.
2. Get your authentication credentials
Once on this page, create a project and answer all the questions. Then you will get the following page with all the credentials. Make sure you don’t share this information with anyone.
Implementation requirements
Install the tweepy module
pip install tweepy # install the tweepy module
Import the required modules
import tweepy # tweepy module to interact with Twitterimport pandas as pd # Pandas library to create dataframesfrom tweepy import OAuthHandler # Used for authenticationfrom tweepy import Cursor # Used to perform pagination
Python implementation
Implement utility functions to get tweets:There are three main functions in this example. (1) An authentication function. (2) A client function in order to interact with the Twitter API. (3) A final function that collects tweets and creates a data frame containing some specific information about a given Twitter account. The information collected are the following: > the date the tweet was created > the author of the tweet> the screen name corresponding to the name on the Twitter account> number of likes obtained by that tweet> number of retweets from the tweet
"""Twitter Authentification CredentialsPlease update with your own credentials"""cons_key = ''cons_secret = ''acc_token = ''acc_secret = ''# (1). Athentication Functiondef get_twitter_auth():    """    @return:        - the authentification to Twitter    """    try:        consumer_key = cons_key        consumer_secret = cons_secret        access_token = acc_token        access_secret = acc_secret            except KeyError:        sys.stderr.write("Twitter Environment Variable not Set\n")        sys.exit(1)            auth = tweepy.OAuthHandler(consumer_key, consumer_secret)    auth.set_access_token(access_token, access_secret)        return auth# (2). Client function to access the authentication APIdef get_twitter_client():    """    @return:        - the client to access the authentification API    """    auth = get_twitter_auth()    client = tweepy.API(auth, wait_on_rate_limit=True)    return client# (3). Function creating final dataframedef get_tweets_from_user(twitter_user_name, page_limit=16, count_tweet=200):    """    @params:        - twitter_user_name: the twitter username of a user (company, etc.)        - page_limit: the total number of pages (max=16)        - count_tweet: maximum number to be retrieved from a page            @return        - all the tweets from the user twitter_user_name    """    client = get_twitter_client()        all_tweets = []        for page in Cursor(client.user_timeline,                         screen_name=twitter_user_name,                         count=count_tweet).pages(page_limit):        for tweet in page:            parsed_tweet = {}            parsed_tweet['date'] = tweet.created_at            parsed_tweet['author'] = tweet.user.name            parsed_tweet['twitter_name'] = tweet.user.screen_name            parsed_tweet['text'] = tweet.text            parsed_tweet['number_of_likes'] = tweet.favorite_count            parsed_tweet['number_of_retweets'] = tweet.retweet_count                            all_tweets.append(parsed_tweet)        # Create dataframe     df = pd.DataFrame(all_tweets)        # Revome duplicates if there are any    df = df.drop_duplicates( "text" , keep='first')        return df
All the functions are finally implemented, now we can continue with data collection. Let’s say we want to collect tweets from Google AI. The function creating the data frame takes the Twitter username/screen name as a mandatory parameter, in this case, it is GoogleAI, without the @ sign.
googleAI = get_tweets_from_user("GoogleAI")print("Data Shape: {}".format(googleAI.shape))
The print instruction shows Data Shape: (1743, 6) meaning we have 1743 tweets from Google AI, and the following.head(10) gives the first 10 tweets in the data frame
googleAI.head(10)
Advantages
Well written documentation, with a very active community
Provides many features about a given tweet (e.g. information about a tweet’s geographical location, etc.)
Disadvantages
Limits the user to the last 3200 tweets in a timeline.
In this article, you have learned how to get your Twitter developer credentials, and how to use tweepy to get data from Twitter. Also, you have learned about the limitations and benefits of this tool.
For further readings do not hesitate to consult the following links:
Source code on Github
Twitter Developer Documentation
Tweepy documentation
Twint documentation
How to get access to the Twitter API
Using Twitter for data collection with health-care consumers: a scoping review
Bye for now 🏃🏾