Developing Good Twitter Data Visualizations using Matplotlib | by Arief Anbiya | Towards Data Science

In this article, we will learn about how to collect Twitter data and create interesting visualizations in Python. We will briefly explore about how to collect tweets using Tweepy and we will mostly explore about the various Data Visualization techniques for the Twitter data using Matplotlib. Before that, Data Visualization and the overall statistical process that enables it will be explained.
First, what is Data Visualization? It is a pictorial or visual representation of data. We have seen it in newspapers, news media, sports analysis, research papers, and sometimes in ads. The common examples of data visualization models are: Line Graph, Scatter Plot, Pie Chart, and Bar Chart (it is easy to make visualizations using these models).
Creating Data Visualization models is important, why?- A picture can speak a thousand words. A graph that represents data of many rows can give a big picture of the data and can reveal patterns contained in the data.- Data visualization is part of statistical analysis. Statistics has many applications in various areas.- Developing new algorithms can train your creativity and problem-solving abilities. Matplotlib has its own tools for plotting data, but we may not restrict ourselves to it in order to have more models and more variety for the visualizations.- Different models can give different perspectives about the data.
The statistical process (P-C-A-I)The process follows a simple flow. The flow starts with Posing the question, for example: “What do @NatGeo account tend to post?”. After we have the question in mind, we perform appropriate data Collection. In the collection phase, perhaps there are many ways to obtain Twitter data, but here pick to use Tweepy. After the collection we will enter the Analysis part. This is where we should choose the appropriate mathematical or statistical method for analyzing the data. One way of analyzing the data is through data visualization. Finally, the last phase is to Interpret the result. Notice that we may end up getting another question to answer, which means the flow could be cyclic.
We have briefly discussed data visualization and the statistical process that enables it. For our investigation, we already know that the questions to answer will be related to Twitter data, hence we will see the data collection first before defining the questions specifically.
Collecting Twitter dataWe can collect a relatively large Twitter data automatically using the Tweepy library in Python. It is an external Python library for accessing Twitter API. In order to make the library useful, we must first create a Twitter Application for our Twitter account. After the registration, we will obtain private information: consumer key, consumer secret, access secret, and access token. The information is required to access our Twitter API using Tweepy.
An example of creating our Twitter API object in Python:
import tweepyconsumer_key = 'ecGxfboL66oO2ZwxfKkg7q3QK'consumer_secret = 'exVRiv517gdwkPLP19PtlQMEIRjxgJr21JZCAAQYIqJCUW5vmh'access_token = '3151279508−Ywd662Zv97Ie7E7I97dUm0e3s2X8yYBloJQd6Gr'access_secret = 'BH5REW4V7RdGadMr31NLY9ksFypG12m8BR04S32ZF7jO3'auth = tweepy.OAuthHandler(consumer_key, consumer_secret)auth.set_access_token(access_token, access_secret)our_api = tweepy.API(auth)
Now we can start using the Twitter API through our_api .
We can directly collect tweets from our home timeline by applying the method our_api.home_timeline() , which collects 20 most recent tweets by default (including retweeted tweets). To adjust the desired number of tweets (take 100 tweets for example), use our_api.home_timeline(count = 100). To collect tweets from a particular account (take @NatGeo for example), use the method our_api.user_timeline(screen_name = 'NatGeo', count = 100).
One weakness of the above methods is that it can only collect up to 200 tweets. To overcome this, we may use the tweepy.Cursor class. To collect 2000 @NatGeo tweets and then saving the texts, number of retweets, and number of likes inside a list, we can perform as below.
natgeo_cursor = tweepy.Cursor(our_api.user_timeline, screen_name = 'NatGeo')natgeo_tweets = [(tweet.text, tweet.retweet_count, tweet.favorite_count) \                 for tweet in natgeo_cursor.items(2000)]
Each item in the natgeo_cursor.items(n)is a tweepy.models.Statusobject from which we can obtain the tweet data such as text, author information (in the form of tweepy.models.User object), time of creation, number of retweets, number of likes, and media.
Result:
>>> natgeo_tweets[0]("As Chernobyl's gray wolf population increases, their influence on the surrounding environment is called into questi... https://t.co/elsb4FJQGZ", 428, 1504)>>> natgeo_tweets[1]('1,058 temples in the south Indian state of Kerala that have pledged to eliminate plastic this year https://t.co/ltJ6mFIWpV', 268, 985)
So far, we have seen how we can use Tweepy to collect actual Twitter data. Now let’s be curious and visualize them using Matplotlib.
*Instead of the above result, we will use various sets of Twitter data for the visualization.
Minimum Data VisualizationTo start this, let us see an example of plotting the username frequency of tweets in our home timeline. The username frequency is the number of tweets from a specific username that appears in our data. The tweets data will be of 200 tweets from the home timeline. Below is the code sample followed by the bar plot result.
import matplotlib.pyplot as plthome_cursor = tweepy.Cursor(api.home_timeline)tweets = [i.author.screen_name for i in home_cursor.items(200)]unq = set(tweets)freq = {uname: tweets.count(uname) for uname in unq}plt.bar(range(len(unq)), freq.values())plt.show()
As you can see, the plot shows that there were 42 different Twitter accounts tweeting in the timeline, and there is only one account that tweets the most (41 tweets). In case you want to know, the most frequent account is @CNNIndonesia. But we have to admit that we cannot see the usernames that appeared in the data, there is not much information can be collected from the bar chart (although it is still better than no visualization at all). That is why we should learn to develop our own algorithms for modifying data visualizations on top of Matplotlib (which can lead to various models).
What can we learn from the attempt above?According to David McCandless, the author of www.informationisbeautiful.net, there are 4 key elements that make a good data visualization: Information, Story (concept), Goal, and Visual Form (metaphor). Information is data, the keywords for this element are accuracy, consistency, and honesty. Story (concept) is to make the data vis interesting. Goal is the usefulness and efficiency of the data vis. Visual Form (metaphor) is the beauty and appearance.
The first bar chart above surely has the Information element, the data is accurate. It also has a bit of Goal and minimum Visual Form (the bars). But it does not have Story (concept) at all (only the creator of the data vis that knows the story behind the plot), it should at least inform the viewers that it is about Twitter data.
To improve our visualization, we will apply the 4 key elements. But to make it interesting, we will only use Matplotlib in Python.
In the next sections, we will see some questions or objectives with respect to various Twitter data and also perform the data visualizations.
Case 1. Objective: accounts comparison, Model: modified horizontal bar chartIn this case, we will use a data of 192 tweets that I have collected from my home timeline a few months ago. The data will be transformed such that we can group the tweets by accounts, and then we will compare the numbers through a data visualization. It is important to note that we will develop a good data visualization that combines various Matplotlib features and not restrict ourselves to the plotting models it has.
The modules that we will use in Python are:
import numpyimport matplotlib.pyplot as pltimport matplotlib.font_manager as fmimport operatorimport itertools
The code for data representation is shown below. It will group the tweets by username and then count the tweets count (also can be seen as username frequency) and the number of followers for each username. It will also sort the data by the username frequency.
home_tweets = list(numpy.load('testfile.npy'))unicity_key = operator.attrgetter('author.screen_name')tweets = sorted(home_tweets, key=unicity_key)authors_tweets = {}followers_count = {}sample_count = {}        for screen_name, author_tweets in itertools.groupby(tweets, key=unicity_key):    author_tweets = list(author_tweets)    author = author_tweets[0].author    authors_tweets[screen_name] = author    followers_count[screen_name] = author.followers_count    sample_count[screen_name] = len(author_tweets)maxfolls = max(followers_count.values())sorted_screen_name = sorted(sample_count.keys(), key = lambda x: sample_count[x])sorted_folls = [followers_count[i] for i in sorted_screen_name]sorted_count = [sample_count[i] for i in sorted_screen_name]
Next, we could easily perform the horizontal bar plot byplt.hbar(range(len(sorted_count)),sorted_count)and then setting the titles for the figure and x-y axis, but let us improvise. We will add additional visual form by adding color transparancy difference, for example: the transparancy of the bars may represent the number of followers of the accounts. We will also change the tick labels on the y axis from numbers to strings of usernames. Here is the code:
colors = [(0.1, 0.1, 1, i/maxfolls) for i in sorted_folls]fig, ax = plt.subplots(1,1)y_pos = range(len(colors))ax.set_yticks(y_pos)ax.set_yticklabels(sorted_screen_name, font_properties = helv_8)ax.barh(y_pos, sorted_count, \        height = 0.7, left = 0.5, \        color = colors)for i in y_pos:    ax.text(sorted_count[i]+1, i, style_the_num(str(sorted_folls[i])), \            font_properties = helv_10, color = colors[i])ax.set_ylim(-1, y_pos[-1]+1+0.7)ax.set_xlim(0, max(sorted_count) + 8)ax.set_xlabel('Tweets count', font_properties = helv_14)plt.tight_layout(pad=3)fig.show()
Notice that we also change the font style, and improvise for the strings of followers count (mapping 22441917 to 22,441,917) using the style_the_num function. Below is the plot result:
It is clear that the plot above shows more information and looks better than the first plot. The visual effects also help to explain the data. The plot also has more value on the Story (concept) element, it looks quite alive (we don’t need to put much effort to understand the visualization).
*Sorry, but maybe the plot should have a text explaining the meaning of the blue color transparency (which is the number of followers of the accounts).
Case 2. Objective: the variety of images posted by an account and the audience’s preference, Model: Image Plot In this case, we will dataset of 1000 @NatGeo tweets (collected on 19 Sept 2018). Each of the 1000 tweets can either has an image or not. So to achieve the objective of this case, we must first collect all images that exist in the data. After that, an Image plot will be introduced. The Python implementation is done by creating PlotDecoratorclass, then use the instance to generate the image plot. The PlotDecoratorclass has methods gen_img(prev_img, next_img), put_below(left_img, next_img), and generate(imgs, ncol). An explanation for the algorithm:
- gen_img(prev_img, next_img):It plots the next image to the right side of the previous image tightly. The next_imgparameter is a matplotlib.image.AxesImage object, while the prev_img is a dictionary containing AxesImageobject of the left image. The function sets tje next image properties so that it will be positioned next to the previous image (the left side one) using the next_img.get_extent(...)and next_img.set_extent(...)methods.
- put_below(left_img, next_img):Similar as the 1st method explained above, but the next image is positioned below the leftest image of the row. This is applied because each row can only have ncol number of images.
- generate(imgs, ncol):This is where we generate the new position of the images. imgsis a list of AxesImageobjects that we want to plot. For example, if ncol=10, this function will apply gen_img(prev_img, next_img)10 times, and then apply put_below(left_img, next_img)and then start over again for the new row of images.
After collecting and positioned the images neatly, we count the number of retweets of each image and then only show auxiliary decoration for the top 3 with highest retweet count. The decoration is only a purple circle with no fill surrounding the image (the transparency gives clue about the rank). Here is the result, it starts from full image, then zoomed in, and then zoomed in again for the most retweeted image:
Let us rate the data visualization above. It is clear that it has a decent value (not bad) of Visual form (metaphor) and the concept is quite nice (it is interesting). The information we get are the top 3 images on highest retweet count and the pattern of the images (all images are great photos, there is no images of posters, quotes, etc).
*Again, maybe the plot should have a short paragraph that explains the meaning of the plot.
Case 3. Objective: the relation between two words, Model: Twitter Venn diagramIn this case, we will use a dataset of @jakpost (The Jakarta Post) 3214 tweets. The oldest tweet is created on 8 May 2018, and the newest is created on 22 June 2018. The objective is that we want to know whether or not there is a relation between two words in a set of tweets. One way to find out is to check whether the picked words are used in one tweet.
There is a method called Twitter Venn diagram that can be used to visualize how two words are connected in a set of tweets. We will adopt the Twitter Venn diagram and improvise a little bit using Matplotlib (the code example can be reached in this repo).*I used TextBlob to collect the words from the tweets.*The algorithm is written in the most obvious way without any optimization attempt.
The improvisation: Each marker representing each tweet has visibility value which represents the retweet count ratio (among the tweets in the circles, the tweet with the most retweet count will have the highest visibility value, which is 1).
Now let us pick two interesting words, for example, ‘jakarta’ and ‘administration’. Tweets that include ‘jakarta’ but not ‘administration’ will have red markers, and tweets that include ‘administration’ but not ‘jakarta’ will have blue markers. Tweets that include both words will have green markers. Below is the plot.
The above plot should have speak about the meaning itself. Among the 3214 tweets, 161 tweets include ‘jakarta’, only 14 tweets include ‘administration’, and 9 tweets include both. The retweets are gathered in the ‘jakarta’ circle, notice that there are 3 tweets with very high retweet count.
Now let us take another example, with words ‘jokowi’ and ‘prabowo’. ‘jokowi’ will be red and ‘prabowo’ be blue. Below is the plot.
If you are curious about the one tweet in the intersection, here is what it says:“ Prabowo accuses Jokowi govt of weakening TNI #jakpost https://t.co/BxJ7hum7Um “
In general, the above Twitter data visualization examples are not the best examples of visualization. But we have seen how we can harness Tweepy and Matplotlib while putting the 4 elements for good data visualization in mind. We have seen how we can combine different colors and shapes to add more values for the visual form (metaphor) and the story (concept).
Some of great data visualization examples (in general, not limited to Python or Matplotlib) can be seen in the links below.- https://informationisbeautiful.net/- https://datavizproject.com/