Hands-on Web Scraping: Building your Twitter dataset with python and scrapy | by Amit Upreti | Towards Data Science

I get it — You are tired of searching for datasets online for your machine learning project or maybe for analyzing a popular Twitter trend.
Today we will learn how to generate your own custom dataset from Twitter by using hashtag search. According to internetlivestats.com, every second, on average, around 6,000 tweets are tweeted which corresponds to over 350,000 tweets sent per minute and 500 million tweets per day. This makes Twitter an excellent place to get data for your projects as Tweets are an accurate representation of today’s natural language on social media.
Want to skip the post and see the good stuff directly? Here is the Github repo for you
First things first, let’s make sure we follow the policies made by twitter.com for robots to follow while crawling so that we don’t run into any legal issues. This can be found at https://twitter.com/robots.txt
We can see from the User-agent and Allow section(lines 1–4) that Twitter allows all the robots to visit the hashtag and search URLs with a 1-second delay between the crawl requests.
User-agent: *Allow: /hashtag/*?src=Crawl-delay: 1
This assumes that you have some basic knowledge of python and scrapy. If you are interested only in generating your dataset, skip this section and go to the sample crawl section on the GitHub repo.
For searching for tweets we will be using the legacy Twitter website. Let’s try searching for #cat https://mobile.twitter.com/hashtag/cats. We want to use this legacy version for gathering tweets URLs as it loads the data without using javascript which makes our job super easy.
I choose a lazy person to do a hard job. Because a lazy person will find an easy way to do it.
― Bill Gates
Finding all the tweet URL for our hashtag search — find_tweets( ) function
find_tweets function for scrapy crawler
Explanation: We will find all the tweets URL in the current page using XPaths and crawl these URLs and send the response to our 2nd function parse_tweet ()
tweets = response.xpath('//table[@class="tweet  "]/@href').getall()    logging.info(f'{len(tweets)} tweets found')    for tweet_id in tweets:            tweet_id = re.findall("\d+", tweet_id)[-1]            tweet_url = 'https://twitter.com/anyuser/status/'+str(tweet_id)               yield scrapy.Request(tweet_url, callback=self.parse_tweet)
Now we will find the next URL for Load older Tweets button and crawl it and send the response to our current find_tweets() function. This way our crawler will keep clicking on the Load older Tweet button recursively if it is available on every crawl. This means we will visit all the results pages one by one.
next_page = response.xpath(    '//*[@class="w-button-more"]/a/@href').get(default='')logging.info('Next page found:')if next_page != '':    next_page = 'https://mobile.twitter.com' + next_page    yield scrapy.Request(next_page, callback=self.find_tweets)
Finding all the data from an individual tweet — parse_tweet( ) function
Explanation: Here, we will use the current version of Twitter (example: https://twitter.com/catsfootprint/status/1213603795663491075) as it turns out the current version loads all the data such as no of likes, retweets, comments, and other metadata. Here is a comparison of loading the above tweet in legacy and the current version of twitter.
parse_tweet( ) function
Our dataset will consist of 14 columns which consist of almost all of the data that can be scraped from a tweet. At the moment the crawler doesn’t collect the comments. The dataset will also include the columnshashtags and mentions which are obtained by parsing the tweet text and searching for words with # at the beginning(for finding hashtags) and @ at the beginning (for finding mentions).
A few ways you might want to use this dataset is:
Sentiment analysis for a popular twitter trend
Competitors Analysis (For example one could find and analyze tweets of their competitors)
Build your Database of tweets by hashtags
Analyze a popular trend. For example, one could find and analyze tweets with #globalwarmingto to find out what the opinion of people.
At the moment, we can see that Twitter allows crawling of its content search results — i.e., tweets and hashtags, and that there is no restriction on how many pages we can crawl. Therefore, our crawler is not affected by any kind of rate limit, and we are able to crawl millions of tweets with this simple script using hashtags and tweet search method. To collect a large number of tweets, I would recommend you to use hundreds of hashtags and run the crawler on a VPS server or scrapy cloud to avoid any kind of interruption.
If you have suggestions or find some issues. I welcome you to open an issue or a Pull Request on GitHub.