What Python package is best for getting data from Twitter? Comparing Tweepy and Twint. | by JP Hwang | Towards Data Science

Twitter is an absolute treasure trove for any data scientist, whether they be a professional, student or hobbyist. It connects people from all walks of life, whether you are an aspiring musician or Taylor Swift, whether you are a rec-league basketball player or LeBron James, all the way from local politicians to the current and past presidents of the United States.
How broad is its user base, and how much data does it generate? As of Q3 2019, Twitter’s daily active user base was about 145 million people, and in 2018 half a billion tweets were being sent out every day. Even with its 280-character limit, it is no surprise that Twitter data presents a fertile ground from which data insights can be harvested.
But then — how do we collect some of this data? In fact, for various reasons — there are limited, pre-compiled repositories of Twitter data (see this link for a discussion). Twitter itself on special occasions might make its own compilations of data available, they are few and far between. They might even not be publicly available — only for a select group of developers and researchers, like this COVID-19 related dataset.
That’s why I wanted to spend some time in this post comparing two popular third party Python packages, Tweepy and Twint.
Both of these are tremendously popular repositories, as you see in the graph of its GitHub popularity below.
I built my own Twitter data analysis app here http://tweetdash.jphwang.com, for which I tried gathering data with both libraries.
As with any tools, they’re not perfect, and each of Tweepy and Twint have strengths and weaknesses in my view. But, they are excellent tools and even with my limited experience, I think most of your twitter data needs can be met with one or a combination of these libraries.
Without much further ado, let’s get started!
So the biggest difference to my mind is this — Tweepy is a Python library for accessing the official Twitter API — as such, you need to apply for developer access and get keys from Twitter.
Link to apply for Twitter developer access
Link showing how to get Twitter API keys
On the other hand, Twint is a scraping tool that is designed to grab tweets and overcome limitations of the API.
(Note: I will not get into the discussion about whether web scraping is legal, or what Twitter’s TOS says about scraping. This is just a discussion about each tool.)
Tweepy’s documentation is in my view, top-notch. Its documentation includes tutorials for everything from authentication to streaming, and includes an API reference for all of its methods.
Starting from zero experience, you should be able to get started with Tweepy in no time to download timelines, user information or perform a search in no time.
On the other hand, browsing Twint’s documentation gives you a feeling that the package has outrun the documentation, or maybe they just haven’t gotten around to it.
Interestingly, Twint is usable from the shell command line — so if the user prefers to, they can simply run twint without going into a Python shell or writing a Python script at all.
What does that all mean? Practically, if the only thing that you are looking to do is collect a large number of tweets, Twint is probably a better tool, whereas Tweepy is better suited for collecting a richer set of metadata, allows for flexibility and potentially scalability as well for those using the official API.
That’s probably still a little abstract. Let’s move on to look at some more concrete examples.
Once you’ve set up your developer account, and your keys & tokens — this code snippet will get you a set of tweets — in this case, I am grabbing the 200 latest tweets from LeBron James’ twitter account.
import tweepyauth = tweepy.OAuthHandler(consumer_key, consumer_secret)auth.set_access_token(access_token, access_token_secret)api = tweepy.API(auth)screenname = 'KingJames'tweets = api.user_timeline(screenname, count=200)
To do the same with Twint, you can actually just run this command from the command line/shell:
twint -u KingJames --limit 200 > testtwint.txt
And to do this in Python, something like this should work:
import twintc = twint.Config()c.Limit = 200c.Store_csv = Truec.Username = usernamec.Output = fnametwint.run.Search(c)
So that’s both relatively simple enough. And they both include more or less all that you might want to know about each tweet.
Tweepy produces data with these columns:
['created_at', 'id', 'id_str', 'full_text', 'truncated', 'display_text_range', 'entities', 'extended_entities', 'source', 'in_reply_to_status_id', 'in_reply_to_status_id_str', 'in_reply_to_user_id', 'in_reply_to_user_id_str', 'in_reply_to_screen_name', 'user', 'geo', 'coordinates', 'place', 'contributors', 'is_quote_status', 'retweet_count', 'favorite_count', 'favorited', 'retweeted', 'possibly_sensitive', 'lang']
On the other hand, Twint produces data with these columns:
['id', 'conversation_id', 'created_at', 'date', 'time', 'timezone', 'user_id', 'username', 'name', 'place', 'tweet', 'mentions', 'urls', 'photos', 'replies_count', 'retweets_count', 'likes_count', 'hashtags', 'cashtags', 'link', 'retweet', 'quote_url', 'video', 'near', 'geo', 'source', 'user_rt_id', 'user_rt', 'retweet_id', 'reply_to', 'retweet_date', 'translate', 'trans_src', 'trans_dest']
Yes, it’s a lot of columns. It’s confusing. But the good news is that the two sets are more or less identical. They kinda have to be, as they are both in theory downloading the same data from Twitter.
In my experience, the time taken to grab tweets were really more or less inconsequential in the scheme of any text processing. Twint in my experience is a tad slower — which makes sense to me, as it is leveraging Twitter’s search function for its scraping, rather than going through Twitter’s native API.
Which takes me to my next point...
The biggest limitation of Tweepy is that of the Twitter API. Twitter’s API has various limits depending on your account tier (pricing), and on top of that, Twitter’s API limits you to the last 3200 tweets in a timeline.
On the other hand, Twint’s main limitation is that it is really only designed to grab tweets, and that’s it. If you want to interact with Twitter — post tweets, block/mute people, send DMs, etc — that’s all Twitter API territory, and you should turn to Tweepy for all that.
As foreshadowed, I would recommend Tweepy over Twint for pretty much any other use case other than grabbing a bunch of tweets for analysis. What other use cases are there? Here are some examples.
Maybe you’d like to occasionally make sure that you follow back everyone who follows you:
auth = tweepy.OAuthHandler("consumer_key", "consumer_secret")redirect_user(auth.get_authorization_url())auth.get_access_token("verifier_value")api = tweepy.API(auth)for follower in tweepy.Cursor(api.followers).items():    follower.follow()
Perhaps you’d like to build a bot that would retweet automatically when its name is mentioned — you’re in luck, for this bot has been built here (by Charles Hooper).
Or maybe you’d like to follow this tutorial on RealPython., and learn how they go about making apps that can automatically fav or retweet according to your wishes.
Not only does the Twitter API allow for a huge number of examples, there are also numerous examples to follow to build your own apps. This simply isn’t possible with Twint. Twint more or less does one thing and does it well. But that’s about it.
Remember how I mentioned that I built my own Twitter data analysis app (http://tweetdash.jphwang.com)? Well, my experiences were that Twint was a great tool for getting the data for building this demo app.
On the other hand, the process of learning and using Twint was incredibly painful, due to its minimal documentation. When I tried to do anything that wasn’t shown in the example code, or deviate from the code example, it required orders of magnitude more time than I did with Tweepy.
With Tweepy, I could rely on its official documentation, other people’s tutorials or example projects. As ever, your mileage may vary, but that’s just been my experience.
All this is to say that use Tweepy as much as possible — and only when you hit a wall in getting raw tweets, see if Twint can be a substitute. I think you’d save yourself a lot of headache that way, and rarely are you going to need more data than the API would allow.
That’s all for today.
If you liked this, say 👋 / follow on twitter, or follow here for updates. In case you missed it, check out this article on building web apps with Plotly Dash:
towardsdatascience.com
And this one comparing financial market APIs:
towardsdatascience.com
I’ve taken a lot of time off from writing because of personal stuff going on, but I’m happy to be back here. :) See you next time!