How to Scrape More Information From Tweets on Twitter | by Martin Beck | Towards Data Science

Sometimes only scraping text and using simple search queries on tweets is not enough. After receiving a fair amount of inquiries about topics not fully covered in my basic Twitter scraping article, I decided it was worth writing a follow-up article that explains the many attributes and methods offered with Tweepy and GetOldTweets3 to further refine one’s search queries for tweets. Hopefully, this article will cover any lingering questions or provide new ideas. Like my previous article, I’ll be using Tweepy, a python library for accessing Twitter’s API, and Dmitry Mottl’s GetOldTweets3, a python library to scrape data without requiring API keys.
This tutorial is meant to be a straightforward article that jumps right into the coding, showcasing the data available in the various objects from each library and methods to further refine one's query when they’re scraping tweets.
Due to the interest in a non-coding solution for scraping tweets, my team is creating an application to fulfill that need. Yes, that means you don’t have to code to scrape data! We are currently in Alpha testing for our app Socialscrapr. If you want to participate or be contacted when the next testing phase is open please sign up for our mailing list below!
We're so pumped to have you!

Already have an account? Sign In

This article assumes you have familiarity with Python and either Tweepy or GetOldTweets3. If this is your first time scraping with these libraries or you need a refresher, I recommend reading my previous article that covers the basics and setting up.
Before we start, a quick overview of the two libraries I’ll be using in this article.
Tweepy:
Can interact with Twitter’s API and create tweets, favorite, etc.
Requires signing up to receive credentials
Access to deeper tweet information and user information
Has limitations in regards to scraping a large number of tweets and access to tweets older than a week using the standard search API
GetOldTweets3
UPDATE: DUE TO CHANGES IN TWITTER’S API GETOLDTWEETS3 IS NO LONGER FUNCTIONING. SNSCRAPE HAS BECOME A SUBSTITUTE AS A FREE LIBRARY YOU CAN USE TO SCRAPE BEYOND TWEEPY’S FREE LIMITATIONS. MY ARTICLE IS AVAILABLE HERE FOR SNSCRAPE.
Only used for scraping tweets
No signup required
Limited access to user information
No restrictions for scraping a large number of tweets or accessing older tweets
Those are the major characteristics of the two libraries, I provide a more detailed comparison in my previous article if you’re interested. If you find yourself wanting Tweepy level access to information on large amounts of data I recommend looking into Twitter’s Premium/Enterprise Search APIs. This article does a great job of showcasing a library called searchtweets that allows you to use these APIs with Python. If you find Twitter’s Premium/Enterprise APIs too expensive, there’s a workaround where you can use Tweepy and GetOldTweets3 together that I discuss at the end of this article.
This article covers different topic niches and because of that, I don’t recommend reading start to end unless you absolutely want a deep understanding of both Tweepy and GetOldTweets3. As such, I’ll include a linked table of contents so you can decide which information is relevant to you and jump to the respective section. Also, as you will see, most of my functions are set to scrape for 150 tweets. I recommend testing your queries on smaller samples first, then scraping your desired tweet amount as large queries may take time to finish.
Quick note, if you’re scraping private accounts or previously private accounts you may find that you can not access the tweets or may be limited in the number of tweets you can scrape.
If you want to check out my code beforehand or have something to follow along with while reading each section I have created an article notebook that follows the code examples presented in this article; and a companion notebook that provides a lot more examples to better flesh out the code. You can access my Jupyter Notebooks for this tutorial on my GitHub here.
Without further ado let’s jump into the scraping!
Getting More Information From Tweets, e.g. favorites, retweets
Getting User Information From Tweets, e.g. follower count, tweet counts
Scraping With Advanced Queries, e.g. scraping from specific locations, or in specific languages
Putting It All Together
Getting More Information From Tweets, e.g. favorites, retweets
Getting User Information From Tweets, LIMITED
Scraping With Advanced Queries, e.g. scraping from specific locations, or in specific languages
Putting It All Together
If you want code to follow along with this section I have an article Jupyter Notebook here. If you want more code examples and easy to use functions I’ve created a companion Jupyter Notebook here.
Before you can utilize Tweepy it requires authorization. I won’t go into detail about setting this up since I’ve covered it before in my previous article.
consumer_key = "XXXXX"consumer_secret = "XXXXX"access_token = "XXXXX"access_token_secret = "XXXXXX"auth = tweepy.OAuthHandler(consumer_key, consumer_secret)auth.set_access_token(access_token, access_token_secret)api = tweepy.API(auth,wait_on_rate_limit=True)
The picture provided below, while not an exhaustive list, covers a majority of information available in Tweepy’s tweet object. If you’re curious about what other data is available, the full list of information is shown here on Twitter’s developer site. As you’ll see in this section and in the user information section. I use string versions of Ids instead of integer in order to keep data integrity since storing the integer value may lead to data loss when accessing it at a later date.
For the most part every tweet.attribute will have some value and are easily accessible. However, the tweet.coordinates and tweet.place attributes require some extracting because they’re data dictionaries that contain other data. Also, because they can be null it’s important to check first if a tweet has the information in the first place. The functions below check if the attribute has data and extracts the relevant information. This code assumes you’re following my column naming convention as we’ll see in the next section. If you change how you name Tweet Coordinates or Place Info you’ll need to adjust the code below accordingly.
# Function created to extract coordinates from tweet if it has coordinate info# Tweets tend to have null so important to run check# Make sure to run this cell as it is used in a lot of different functions belowdef extract_coordinates(row):    if row['Tweet Coordinates']:        return row['Tweet Coordinates']['coordinates']    else:        return None# Function created to extract place such as city, state or country from tweet if it has place info# Tweets tend to have null so important to run check# Make sure to run this cell as it is used in a lot of different functions belowdef extract_place(row):    if row['Place Info']:        return row['Place Info'].full_name    else:        return None
Now that we can move forward and be able to extract useful information from the tweet.coordinates and tweet.place attributes. Let’s jump into actually scraping.
Example of a search query pulling all tweet related information from a user’s tweets:
username = 'random'max_tweets = 150 tweets = tweepy.Cursor(api.user_timeline,id=username).items(max_tweets) # Pulling information from tweets iterable object# Add or remove tweet information you want in the below list comprehensiontweets_list = [[tweet.text, tweet.created_at, tweet.id_str, tweet.user.screen_name, tweet.coordinates, tweet.place, tweet.retweet_count, tweet.favorite_count, tweet.lang, tweet.source, tweet.in_reply_to_status_id_str, tweet.in_reply_to_user_id_str, tweet.is_quote_status] for tweet in tweets] # Creation of dataframe from tweets_list# Add or remove columns as you remove tweet informationtweets_df = pd.DataFrame(tweets_list,columns=['Tweet Text', 'Tweet Datetime', 'Tweet Id', 'Twitter @ Name', 'Tweet Coordinates', 'Place Info', 'Retweets', 'Favorites', 'Language', 'Source', 'Replied Tweet Id', 'Replied Tweet User Id Str', 'Quote Status Bool']) # Checks if there are coordinates attached to tweets, if so extracts themtweets_df['Tweet Coordinates'] = tweets_df.apply(extract_coordinates,axis=1) # Checks if there is place information available, if so extracts themtweets_df['Place Info'] = tweets_df.apply(extract_place,axis=1)
The above query pulls 150 tweets from Twitter user @random and utilizes the extract_coordinates and extract_place functions to check if there is information and to extract the relevant information if there is. As shown above there is a lot of information available in the tweet objects. To modify the information you just need to add or remove whatever tweet.attribute you want to the list comprehension shown in tweets_list. In doing so it’s also important to adjust the column names when you’re creating the dataframe tweets_df .
Access Tweet Text up to 280 characters
Tweepy by default only pulls tweet text up to 140 characters for compatibility purposes since this was Twitter’s old tweet character limit. This can be annoying as you’ll likely want all of the information available instead of tweets that are shortened. To remove this restriction you must modify the tweet_mode parameter in tweepy.Cursor. If you are using Tweepy it’s important to have this tweet_mode set equal to ‘extended’ because if you don’t include this in your queries you will always be scraping tweets that are shortened to the original 140 character limit. When Tweepy is set to extended tweet mode this also changes the tweet.text attribute to tweet.full_text instead. So make sure to adjust your code accordingly if you use this extended tweet mode. There is a code snippet below showing the change in tweepy.Cursor and tweets_list to access tweets with more than 140 characters. More information regarding the extended tweet mode is available here in Tweepy’s documentation.
username = 'BillGates'max_tweets = 150 # Creation of query method using parameterstweets = tweepy.Cursor(api.user_timeline, id=username, tweet_mode='extended').items(max_tweets) # Pulling information from tweets iterable object# Add or remove tweet information you want in the below list comprehensiontweets_list = [[tweet.full_text, tweet.created_at, tweet.id_str, tweet.user.screen_name, tweet.coordinates, tweet.place, tweet.retweet_count, tweet.favorite_count, tweet.lang, tweet.source, tweet.in_reply_to_status_id_str, tweet.in_reply_to_user_id_str, tweet.is_quote_status] for tweet in tweets] # Creation of dataframe from tweets_list# Did not include column names to simplify code tweets_df = pd.DataFrame(tweets_list)
User information is where Tweepy excels compared to GetOldTweets3. The picture provided below, while not an exhaustive list, covers a majority of information available in Tweepy’s user object. If you’re curious about what other data is available, the full list of information is available here on Twitter’s developer site.
Example of a search query pulling all user-related information from tweets:
text_query = 'Coronavirus'max_tweets = 150 # Creation of query method using parameterstweets = tweepy.Cursor(api.search,q=text_query).items(max_tweets) # Pulling information from tweets iterable object# Add or remove tweet information you want in the below list comprehensiontweets_list = [[tweet.text, tweet.created_at, tweet.id_str, tweet.user.name, tweet.user.screen_name, tweet.user.id_str, tweet.user.location, tweet.user.url, tweet.user.description, tweet.user.verified, tweet.user.followers_count, tweet.user.friends_count, tweet.user.favourites_count, tweet.user.statuses_count, tweet.user.listed_count, tweet.user.created_at, tweet.user.profile_image_url_https, tweet.user.default_profile, tweet.user.default_profile_image] for tweet in tweets] # Creation of dataframe from tweets_list# Did not include column names to simplify code tweets_df = pd.DataFrame(tweets_list)
The above query searches for 150 recent tweets that contain the word coronavirus. Similar to the other code snippet shown earlier. To modify the information that is available you add or remove any tweet.user.attribute to the list comprehension shown in tweets_list.
Tweepy provides several different methods to refine your queries. The picture below shows a list of search parameters accessible through Tweepy. Again, while this is not the exhaustive list it covers a majority of it. If you want to see more information regarding the search API it is available here on Twitter’s developer site. As you might notice with geocode, Tweepy does not have a method that will take in string versions of cities and search around them. You’ll have to enter a specific pair of latitude, longitude coordinates to restrict querying by geographic location.
Example of a search query using advanced queries:
# Example may no longer show tweets if until_date falls outside # of 7-day period from when you run cellcoordinates = '19.402833,-99.141051,50mi'language = 'es'result_type = 'recent'until_date = '2020-08-10'max_tweets = 150 # Creation of query method using parameterstweets = tweepy.Cursor(api.search, geocode=coordinates, lang=language, result_type = result_type, until = until_date, count = 100).items(max_tweets) # List comprehension pulling chosen tweet information from tweets iterable object# Add or remove tweet information you want in the below list comprehensiontweets_list = [[tweet.text, tweet.created_at, tweet.id_str, tweet.favorite_count, tweet.user.screen_name, tweet.user.id_str, tweet.user.location, tweet.user.url, tweet.user.verified, tweet.user.followers_count, tweet.user.friends_count, tweet.user.statuses_count, tweet.user.default_profile_image, tweet.lang] for tweet in tweets] # Creation of dataframe from tweets_list# Did not include column names to simplify code tweets_df = pd.DataFrame(tweets_list)
The above query pulls 150 recent tweets in Mexico City in Spanish with the latest date being August 10th, 2020. This code snippet is a little different than the other two shown before. In order to refine search parameters, you’ll have to add the different parameters shown in the picture above to tweepy.Cursor(geocode = coordinates, lang=language, etc.) and pass it a variable or hardcode it. That way you can refine your search by either location, language, whatever you want to do.
Great, I’ve seen a lot of separate things, but why does this matter? Whether you want to scrape tweets from a specific user searching for keywords or search for tweets within a 50-mile radius of Las Vegas, NV (Lat 36.169786, Long -115.139858) that have the keyword Coronavirus. Your tweet scraping is only limited by your imagination and the attributes and methods available in Tweepy. Below I’ll show you how easy it is to pick and choose the methods and information you want by showing the query mentioned above.
Example of a query pulling tweet and user information with an advanced query:
text_query = 'Coronavirus'coordinates = '36.169786,-115.139858,50mi'max_tweets = 150 # Creation of query method using parameterstweets = tweepy.Cursor(api.search, q = text_query, geocode = coordinates, count = 100).items(max_tweets) # Pulling information from tweets iterable object# Add or remove tweet information you want in the below list comprehensiontweets_list = [[tweet.text, tweet.created_at, tweet.id_str, tweet.favorite_count, tweet.user.screen_name, tweet.user.id_str, tweet.user.location, tweet.user.followers_count, tweet.coordinates, tweet.place] for tweet in tweets] # Creation of dataframe from tweets_list# Did not include column names to simplify codetweets_df = pd.DataFrame(tweets_list)
As you see above just modify the code in tweepy.Cursor(api.search,) with search parameters to further refine your search by location, top_tweets, etc. If you want to modify the information that you receive from tweets just add or remove a tweet.chosen_attribute in the tweets_list list comprehension I created below the query method.
UPDATE: DUE TO CHANGES IN TWITTER’S API GETOLDTWEETS3 IS NO LONGER FUNCTIONING. SNSCRAPE HAS BECOME A SUBSTITUTE AS A FREE LIBRARY YOU CAN USE TO SCRAPE BEYOND TWEEPY’S FREE LIMITATIONS. MY ARTICLE IS AVAILABLE HERE FOR SNSCRAPE.
GetOldTweets3 only requires a pip install. After importing the library you should be able to utilize it right away. If you want code to follow along with this section I have an article Jupyter Notebook here. If you want more code examples and easy to use functions I’ve created a companion Jupyter Notebook here.
The picture provided below is an exhaustive list of information available to you through GetOldTweets3's tweet object. As you can see there is a fair amount of information that a single tweet contains, and accessing it is not hard at all.
Example of a search query pulling all tweet related information from a user’s tweets:
username = 'jack'count = 150 # Creation of tweetCriteria query object with methods to specify furthertweetCriteria = got.manager.TweetCriteria().setUsername(username)\.setMaxTweets(count)    # Creation of tweets iterable containing all queried tweet datatweets = got.manager.TweetManager.getTweets(tweetCriteria) # List comprehension pulling chosen tweet information from tweets# Add or remove tweet information you want in the below list comprehensiontweets_list = [[tweet.id, tweet.author_id, tweet.username, tweet.to, tweet.text, tweet.retweets, tweet.favorites, tweet.replies, tweet.date, tweet.formatted_date, tweet.hashtags, tweet.mentions, tweet.urls, tweet.permalink,] for tweet in tweets] # Creation of dataframe from tweets_list# Did not include column names to simplify code tweets_df = pd.DataFrame(tweets_list)
The above query pulls 150 recent tweets from twitter user @jack. As shown above, to modify the information available you just add or remove whatever tweet.attribute you want to the list comprehension shown in tweets_list.
GetOldTweets3 is limited in the user information available compared to Tweepy. This library's tweet object only contains a tweet author’s username and user_id. If you want more user information than is available I recommend either using Tweepy for all of your scraping or using Tweepy methods with GetOldTweets3 in order to utilize both libraries to their strengths. To do that I do showcase some workarounds in my “How to Use Tweepy with GetOldTweets3” section below.
The picture provided below is an exhaustive list of methods available to refine your queries through GetOldTweets3.
Example of a search query using advanced queries:
username = "BarackObama"text_query = "Hello"since_date = "2011-01-01"until_date = "2016-12-20"count = 150 # Creation of tweetCriteria query object with methods to specify furthertweetCriteria = got.manager.TweetCriteria().setUsername(username)\.setQuerySearch(text_query).setSince(since_date)\.setUntil(until_date).setMaxTweets(count) # Creation of tweets iterable containing all queried tweet datatweets = got.manager.TweetManager.getTweets(tweetCriteria) # List comprehension pulling chosen tweet information from tweets# Add or remove tweet information you want in the below list comprehensiontweets_list = [[tweet.id, tweet.author_id, tweet.username, tweet.text, tweet.retweets, tweet.favorites,tweet.replies,tweet.date] for tweet in tweets] # Creation of dataframe from tweets list# Did not include column names to simplify code tweets_df = pd.DataFrame(tweets_list)
The above query tries to pull 150 tweets from Barack Obama that say hello between the date range January 1st, 2011, and December 20th, 2016. As shown in the above code snippet, if you want to use any of the methods to create more specific queries all you need to do is add them at the end of TweetCriteria(). For example, I can further refine my search in the above code by adding .setNear(“Washington, D.C.”) at the end of .setMaxTweets() if I want to query tweets made around that area.
Great I can pull tweet information or pull tweets from 2016. Why does it matter? Whether you want to scrape all available information from a specific user searching for keywords or search for top tweets in Washington, D.C. that have the keyword Coronavirus between August 5th, 2020, and August 10th, 2020. Your tweet scraping is only limited by your imagination and the attributes and methods available in this package. Below I’ll show you how easy it is to pick and choose the methods and information you want by creating the query mentioned above.
Example of a query pulling tweet and user information with an advanced query:
text_query = 'Coronavirus'since_date = '2020-08-05'until_date = '2020-08-10'location = 'Washington, D.C.'top_tweets = Truecount = 150 # Creation of tweetCriteria query object with methods to specify furthertweetCriteria = got.manager.TweetCriteria()\.setQuerySearch(text_query).setSince(since_date)\.setUntil(until_date).setNear(location).setTopTweets(top_tweets)\.setMaxTweets(count) # Creation of tweets iterable containing all queried tweet datatweets = got.manager.TweetManager.getTweets(tweetCriteria) # List comprehension pulling chosen tweet information from tweets# Add or remove tweet information you want in the below list comprehensiontweets_list = [[tweet.id, tweet.author_id, tweet.username, tweet.to, tweet.text, tweet.retweets, tweet.favorites, tweet.replies, tweet.date, tweet.mentions, tweet.urls, tweet.permalink,] for tweet in tweets] # Creation of dataframe from tweets list# Add or remove columns as you remove tweet informationtweets_df = pd.DataFrame(tweets_list)
As this last example shows, refining your queries or modifying the tweet data available to you is rather simple. You refine your queries by modifying the tweetCriteria object by adding set methods onto the end and passing in values. You remove or add more information from your tweets by modifying the tweet attributes listed in tweets_list.
What happens if you use GetOldTweets3 but want access to user information? Or you used Tweepy but need access to older tweets. Well, thankfully there’s a workaround where you can use methods that Tweepy has, to access more tweet or user information. Allowing you to use both libraries together.
It’s important to remember that Tweepy’s API always requires one to sign-up to receive credentials, if you need to do so my previous article covers that. Also, these methods are still subject to Tweepy’s request limitations, so I don’t recommend using this workaround on datasets greater than 20k tweets unless you don’t mind letting your computer run for a couple of hours. I’ve personally used these two libraries together with a dataset of 5k tweets and it took me around 1–2hrs to finish running everything. If you don’t have the time to wait or don’t mind paying, it’s worth looking into utilizing Twitter’s Premium/Enterprise APIs. This article covers searchtweets, a library in Python that allows access to the Premium/Enterprise APIs if you decide to go that route.
If you want code to follow along with this section I have a Jupyter Notebook available here.
Credentials
Before you can utilize Tweepy it requires authorization. I won’t go into detail about setting this up since I’ve covered it before in my previous article.
# consumer_key = "XXXXX"# consumer_secret = "XXXXX"# access_token = "XXXXX"# access_token_secret = "XXXXXX"auth = tweepy.OAuthHandler(consumer_key, consumer_secret)auth.set_access_token(access_token, access_token_secret)api = tweepy.API(auth,wait_on_rate_limit=True)
Preparation
Tweepy’s API offers two methods that allow a GetOldTweets3 user to access more information.
api.get_status() takes in a tweet id and returns all information associated with that tweet
api.get_user() takes either a user id or username and returns all information associated with that user
So let’s make sure we have this data available and scrape for tweet.id, tweet.author_id, and tweet.username using GetOldTweets3.
text_query = 'Hello'since_date = "2020-7-20"until_date = "2020-7-21"count = 150 # Creation of tweetCriteria query object with methods to specify furthertweetCriteria = got.manager.TweetCriteria()\.setQuerySearch(text_query).setSince(since_date)\.setUntil(until_date).setMaxTweets(count) # Creation of tweets iterable containing all queried tweet datatweets = got.manager.TweetManager.getTweets(tweetCriteria) # List comprehension pulling chosen tweet information from tweets# Add or remove tweet information you want in the below list comprehensiontweets_list = [[tweet.id, tweet.author_id, tweet.username, tweet.text, tweet.retweets, tweet.favorites, tweet.replies, tweet.date] for tweet in tweets] # Creation of dataframe from tweets list# Add or remove columns as you remove tweet informationtweets_df = pd.DataFrame(tweets_list, columns = ['Tweet Id', 'Tweet User Id', 'Tweet User', 'Text', 'Retweets', 'Favorites', 'Replies', 'Datetime'])
Functions
Alright perfect, we’ve got our hands on some data that has tweet.id, tweet.author_id, and tweet.username. Let’s test out these methods from Tweepy to make sure they work.
api.get_status(1285363852851511301)api.get_user(811267164476841984)api.get_user('realJakeLogan')
As you’ll see with the above code there’s a lot of information that comes in from the requests. It’s also important to note that Tweepy’s tweet objects contain a user object already. If you only need user information from Tweepy you can use either method but if you want Tweepy’s tweet information you’ll have to use the get_status method. Let’s take this a bit further. The code above is only useful for searching one thing at a time, if you have a whole dataset you’ll need to create a function to extract the data.
def extract_tweepy_tweet_info(row):    tweet = api.get_status(row['Tweet Id'])    return tweet.source tweets_df['Tweet Source'] = tweets_df.apply(extract_tweepy_tweet_info,axis=1)
This is great, however, what happens if you want to return more than a single attribute from the tweet object? Maybe you also want tweet.user.location and tweet.user.followers_count. Well, there are two ways to go about that. Create a function to store that data in a list then add all the data to the data frame. Or create a function that will create a series and return it, then use panda’s apply method to use the function on the data frame. I’ll showcase the former as it’s easier to grasp.
# Creation of list to store scrape tweet datatweets_holding_list = [] def extract_tweepy_tweet_info_efficient(row):    # Using Tweepy API to request for tweet data    tweet = api.get_status(row['Tweet Id'])        # Storing chosen tweet data in tweets_holding_list to be used later    tweets_holding_list.append((tweet.source, tweet.user.statuses_count,          tweet.user.followers_count, tweet.user.verified)) # Applying the extract_tweepy_tweet_info_efficient function to store tweet data in the tweets_holding_listtweets_df_test_efficient.apply(extract_tweepy_tweet_info_efficient, axis=1) # Creating new columns to store the data that's currently being held in tweets_holding_listtweets_df_test_efficient[['Tweet Source', 'User Tweet Count', 'Follower Count', 'User Verified Status']] = pd.DataFrame(tweets_holding_list)
Perfect, instead of having to request Tweepy’s API every time you have another column you can send a single request and modify your data frame one time. This is a lot more efficient and will end up saving you a lot of time if you find yourself needing multiple attributes.
That is all for my advanced tutorial of both Tweepy and GetOldTweets3 and my workaround for using Tweepy with GetOldTweets3. Hopefully, most of your questions should be addressed by my previous article or this article. If you have specific questions and can’t find answers on Google, or want to reach out to me. I’m available on LinkedIn.
If you’re interested, sign up for our Socialscrapr mailing list: https://upscri.be/3kcmqx
snscrape scraping article for if you want a GetOldTweets3 substitute: https://medium.com/better-programming/how-to-scrape-tweets-with-snscrape-90124ed006af
Previous basic twitter scraping article: https://towardsdatascience.com/how-to-scrape-tweets-from-twitter-59287e20f0f1
GitHub containing this tutorial’s Twitter scraper’s: https://github.com/MartinBeckUT/TwitterScraper/tree/master/AdvScraper
GitHub containing both my basic and advanced tutorial Twitter scraper’s: https://github.com/MartinBeckUT/TwitterScraper
Various API’s offered with Tweepy: https://developer.twitter.com/en/products/products-overview
Tweepy GitHub: https://github.com/tweepy/tweepy
GetOldTweets3 GitHub: https://github.com/Mottl/GetOldTweets3
Tutorial article for accessing Twitter’s Premium/Enterprise APIs with searchtweets: https://lucahammer.com/2019/11/05/collecting-old-tweets-with-the-twitter-premium-api-and-python/