How to build a dataset from Twitter using Python tweepy | by Angelica Lo Duca | Towards Data Science

In this tutorial, I explain a very simple procedure to extract data from Twitter using the tweepy Python library. The code can be downloaded from my Github repository.
In detail, I will implement a strategy for the extraction of the tweets of the previous day with hashtag #covid-19 and automation for a daily extraction.
You could adapt this code to extract other hashtags. The output dataset looks like the following one:
It contains the text, the favorite counts for the tweet, as well as the number of retweets, and the creation date.
The architecture of the software is composed of the following elements, as described in the next figure:
Twitter App
Get Tweets
Cron job (configured as a crontab)
Firstly, I need to register as a Twitter Developer on the Twitter Developer Web Site. If I own a Twitter account, I can use it. I should click on Apply and follow the wizard. Once created my Twitter developer profile, I can go to the Twitter Dashboard and select projects and apps:
I create a new app:
I follow the wizard. At the end of the wizard, I register the provided secrets in a separate file, called config.py. This file should look as follows:
TWITTER_CONSUMER_KEY = 'PUT HERE YOUR API KEY'TWITTER_CONSUMER_SECRET = 'PUT HERE YOUR API SECRET'
At the end of the wizard, the new app appears on the left under the heading Projects & Apps
I click on the new App and then on the key symbol:
Then I generate the Access Token and Secret and I register them into the config.py file.
TWITTER_ACCESS_TOKEN = 'PUT HERE YOUR ACCESS TOKEN'TWITTER_ACCESS_TOKEN_SECRET = 'PUT HERE YOUR SECRET TOKEN'
I can implement the Get Tweet script as a jupyter notebook and then I can download it as a Python script.
This script extracts all the tweets with hashtag #covid-19 related to the day before today (yesterday) and saves them into a .csv file. I use the tweepy library, which can be installed with the command pip install tweepy.
Firstly, I import the configuration file, called config.py, which must be located in the same directory of this script.
from config import *import tweepyimport datetime
I set up the connection to our Twitter App by using the OAuthHandler() class and its access_token() function. Then I call the Twitter API through the API() function.
auth = tweepy.OAuthHandler(TWITTER_CONSUMER_KEY, TWITTER_CONSUMER_SECRET)auth.set_access_token(TWITTER_ACCESS_TOKEN, TWITTER_ACCESS_TOKEN_SECRET)api = tweepy.API(auth,wait_on_rate_limit=True)
Now I set up dates. I need to set up today and yesterday.
today = datetime.date.today()yesterday= today - datetime.timedelta(days=1)
I search for tweets on Twitter by using the Cursor() function. I pass the api.search parameter to the cursor, as well as the query string, which is specified through the q parameter of the cursor. The query string can receive many parameters, such as the following (not mandatory) ones:
from: - to specify a specific Twitter user profile
since: - to specify the beginning date of the search
until: - to specify the ending date of the search The cursor can also receive other parameters, such as the language and the tweet_mode. If tweet_mode='extended', all the text of the tweet is returned, otherwise only the first 140 characters.
tweets_list = tweepy.Cursor(api.search, q="#Covid-19 since:" + str(yesterday)+ " until:" + str(today),tweet_mode='extended', lang='it').items()
Now I loop across the tweets_list, and, for each tweet, I extract the text, the creation date, the number of retweets, and the favorite count. I store every tweet in a list, called output.
output = []for tweet in tweets_list:    text = tweet._json["full_text"]    print(text)    favourite_count = tweet.favorite_count    retweet_count = tweet.retweet_count    created_at = tweet.created_at        line = {'text' : text, 'favourite_count' : favourite_count, 'retweet_count' : retweet_count, 'created_at' : created_at}    output.append(line)
Finally, I convert the output list to a pandas DataFrame and I store results. The first time I run this script, I use the to_csv() function without any other parameter. From the second time on, I should add the following parameters: df.to_csv(‘output.csv’, mode=’a’, header=False) , in order to append results to the previously stored output and avoid rewriting the header.
import pandas as pddf = pd.DataFrame(output)df.to_csv('output.csv')
Now, I can download the Python code as a .py script by selecting the corresponding menu in jupyter.
The cron job is a way to schedule some repetitive actions. In this example, the cron job is used to download the tweets of the previous day every day.It requires the machine running the cron job to be turned on at the time the cron job is to be run.
If you have a Linux/Mac OS machine, you can follow this procedure to configure the cron job:
open a terminal and type crontab -ethis command opens a text editor to insert, modify or delete a task
each task must contain the time and date of execution. The order is minutes, hour, day of the month, month, day of the week. You can use the * to indicate all values
insert a new line with00 01 * * * python /<path_to_file>/get_tweets.pythis runs the script once a day at one in the morning.
If you have a Windows 10 machine, you can follow the procedure described in this post to configure the cron job.
Now the Twitter extractor is ready! Enjoy it :)
In this tutorial, I have illustrated how to extract tweets through Python tweepy. The procedure requires three-step: Twitter App setup, code writing, and cronjob setup.
The described mechanism can be also used when you want to extract more than one hashtag. You can simply add all the hashtags you want in the query string.
If you wanted to be updated on my research and other activities, you can follow me on Twitter, Youtube and Github.
medium.com
alod83.medium.com
betterhumans.pub
Follow me on Medium
Sign up for my newsletter
Connect on LinkedIn
Follow me on Twitter
Follow me on Facebook
Follow me on Github