What Can We Learn from Elon Musk’s Twitter Graph? | by Günter Röhrich | Towards Data Science

Graph mining allows us to collect data and build a diagram of nodes and edges from any given set of entities. Algorithms like Louvain method or PageRank provide meaningful tools to analyze graphs after being mined and built. This post describes the building process of a graph with just a few lines of Python code.
Graph analysis is both an incredibly exciting and fast-developing field of data analytics. It reaches from community detection to other sophisticated tasks like pattern recognition. Graph analytics can also be seen as one of the powerhouses enabling search engine giants to add additional interconnected result dimensions to your search results (basically they are relying on so called knowledge graphs), relevant daily news to pop up on your phone or buying recommendations in your favorite online shops.
As a side note: Google’s search is based on the former development of Freebase© which is discontinued, but still partially accessible — link here.
While there are numerous algorithms available for analysis, Louvain method and PageRank are probably the ones that come to one’s mind first when thinking of graphs. The Louvain method allows detecting communities in all sorts of networks, whereas PageRank (named after web pages, not its co-inventor and Google co-founder Larry Page) finds the steady state probability (Markov chain) of reaching a node when randomly jumping between nodes throughout the graph — the latter is probably the best-known algorithm in this scientific field as it is regarded as the ground work of Google search.
Another popular analytics approach is looking at graphs which are characterized through a large number of triangle connections. Finding such graphs is a real-world problem headhunters, social media and marketing experts might especially be interested in.
“Triangle counting gained popularity in social network analysis, where it is used to detect communities and measure the cohesiveness of those communities. It can also be used to determine the stability of a graph, and is often used as part of the computation of network indices, such as the clustering coefficient.” — Neo4j
An incredibly well written 6-page-only paper from Stanford University can be found in the APPENDIX at the end of this post. An easy way to integrate triangle count in your program is explained here:
towardsdatascience.com
In order to provide meaningful data for this kind of analysis, this article allows you to follow just a few simple steps to mine a real-world graph from Twitter. The below-shown functions start from the “RootNode” Elon Musk and iterate over his follows, the follows of his follows, etc.
If you wanted to use your own account as a starting point, you need to obtain your starting ID first. This cannot be done through the frontend, but there is an easy script on Twitter’s GitHub (user lookup) to help you out.
For this project I used Elon’s follows. Using follows makes sense because we would like to explore Elon’s network and not someone else’s network of which Elon is just a part. This logic applies to all other nodes as well— we only care about follows, not followers. Be aware that collecting nodes through Twitter takes quite a long time, for this reason the shown graph is only a small part of Elon’s whole network. The “Elon Musk Graph” can be visually explored (link), by simply clicking on the nodes — drag them around to organize them in a better way. Node sizes are characterized as having a higher PageRank value, which is also expressed through the coloring. Darker (blue) colors indicate a high PageRank score, yellow, lighter colors show low score nodes. Nodes can also be expressed through their degree (outdegree) if desired.
Thanks to Prof. Duen Horng Chau (Georgia Tech) for providing the visualization tool. If you are interested in how Argo Lite works (it is using D3), follow the GitHub repository.
Before exploring the chart and/or going into the mining functions please consider the following constraints:
Twitter only allows 15 API calls per 15 minutes. This is an incredibly small number. Starting with 40 users in the initial iteration means, that we can only have 14 users analyzed in this run (1 run is used for the RootNode). So better have your Raspberry Pi up and running for this.There has to be a limit on the users obtained for the RootNode but alsofor the nodes of nodes. This limit should be even smaller than the first one. As a rule of thumb ≤ 5 — otherwise the graph would become extremely large and the API calls would be far too overwhelming.
Twitter only allows 15 API calls per 15 minutes. This is an incredibly small number. Starting with 40 users in the initial iteration means, that we can only have 14 users analyzed in this run (1 run is used for the RootNode). So better have your Raspberry Pi up and running for this.
There has to be a limit on the users obtained for the RootNode but also
for the nodes of nodes. This limit should be even smaller than the first one. As a rule of thumb ≤ 5 — otherwise the graph would become extremely large and the API calls would be far too overwhelming.
This whole thing starts with exploring the Twitter API. I am not the greatest fan of their documentation (especially this v1.0 to v2.0 is sometimes confusing), however their example code on GitHub is great.
To start with your project you need the following two building blocks:
A ready-to-go project on your Twitter developers page — This is required to obtain keys to authorize your API calls later on.The ID of your RootNode — this is the starting point for the script to iterate through all the “follows”
A ready-to-go project on your Twitter developers page — This is required to obtain keys to authorize your API calls later on.
The ID of your RootNode — this is the starting point for the script to iterate through all the “follows”
The code shown below is just a copy from Twitter dev’s GitHub and build the starting point for our program. Additional imports like logging are not necessary, however, may turn out to be useful when it comes to track the progress of larger graphs. Again, as a reminder, Twitter only allows an unsatisfyingly tiny number of API calls — so logging may come in handy at some point.
Given those prerequisites above, I simply packed the boilerplate code into a function which I call for every iteration. The code is basically taking the node identifier (id) we pass to the function and retrieves the full list of the node/person’s followers.
We further need to set limitation parameters. This is a necessity because due to the size of the Twitter network for an average or even highly involved user this mining task would take an incredibly long time.
iterations: This allows us to define how often we take the node’s follows and select those nodes as next iteration starting points. See also below*iteration = 0 simply defines a counter variable.request_count: is required to limit the number of calls. This counter makes sure we’re not hitting the (incredibly tiny) API limitstart_limit defines the number nodes retrieved in the very first iterationlimit defines the number of nodes for any following iteration
iterations: This allows us to define how often we take the node’s follows and select those nodes as next iteration starting points. See also below*
iteration = 0 simply defines a counter variable.
request_count: is required to limit the number of calls. This counter makes sure we’re not hitting the (incredibly tiny) API limit
start_limit defines the number nodes retrieved in the very first iteration
limit defines the number of nodes for any following iteration
* Iterations:0 — [RootNode -> n Users]1 — [RootNode -> n Users -> m Users ], 2 — [RootNode -> n Users -> m Users -> m Users]# where n is the start_limit and m is limit.# You can push this iterative process to any number you like, # just make sure it comes to end within the next decade.
This first block of code is basically only defining the retrieval of users (get_follows) as well as setting parameters for the mining process.
Now the real fun begins as we obtain the first follows of the initial node (RootNode). To emphasize this again, I am using Python’s built in logging library, a solid way of keeping track of the progress and identify problems the program may be confronted with — just be aware that your program might run many hours, days... — so “print” is probably not a good option.
Another important aspect of this function is that we have a close look at propagated nodes. We are using follows_checked to keep track of the nodes we have already visited and remove them from our stack (new_nodes), otherwise we created an infinity loop.
We are basically iterating over a user list and add all elements to the edges and nodes. The main challenge is to keep track of nodes we already used to get user lists. For this reason we keep:
follows_check is just an exclusion list. Every entry in here will no longer be used to call the data retrievalnodes is the list of all nodes, appended during all iterationsnew_nodes: keeps track of nodes we just introducednext_nodes: is just a temporary array we use to compare to the exclusion list.
follows_check is just an exclusion list. Every entry in here will no longer be used to call the data retrieval
nodes is the list of all nodes, appended during all iterations
new_nodes: keeps track of nodes we just introduced
next_nodes: is just a temporary array we use to compare to the exclusion list.
The first iteration is easy, we simply go over all nodes and append the nodes and edges lists. Note that nodes might not be unique in this list. Make sure you prepare to clean your output from duplicates if required for your use case.
The more complex task arises when we see additional “layers” we want to follow. It is crucial here to keep track of the checked nodes and that we store the newly introduced nodes separately. Think through this again.
This short post has demonstrated how you can iteratively collect data and store it in a structure that can be used to build a graph. The key take away is that the age of big data has brought us a variety of new challenges and graphs are one major way to better understand our data.
Graphs can be stored easily, whether in a simple CSV file (for small graphs), in a data base or even distributed across several computers (e.g. Spark GraphX) and can be approached with powerful and proven algorithms to explore their contents.
If PageRank was what you were looking for, I recommend going through this post, it is not too overwhelming and captures the essentials:
towardsdatascience.com
If you are not only interested in the connections between the users, but also what kind of tweets they made, the sentiment of their Tweets or just the content users write about, you might want to look at natural language processing and/or similarity measures. I once created an iterative method to evaluate the importance of certain words in a text, this could be useful for your project:
towardsdatascience.com
{ See you next time and stay safe }
Again, a huge thanks to Prof. Duen Horng (Polo) Chau and all involved fellow students (Georgia Tech) who provide the online graph visualization tool Argo Lite— Prof. Polo is also the one who really sparked my interest in graph analysis.
Georgia Tech #CS 6242 Data and Visual Analytics