Analyzing Twitter Relationships in Python without an API | by Emre Rençberoğlu | Towards Data Science

Social media analysis is one of the hot topics of data science. People like these analyses and interest them because everyone is familiar with this world. Most of our time goes to Twitter, Instagram, Facebook, and some other social media apps.
As a data enthusiast, this topic caught my attention not surprisingly. However, getting access to the official Twitter API is very challenging. Therefore, I searched for another solution and found out twint. This is a python library that enables you to scrap twitter data without API access.
In this article, I will briefly explain how to scrape twitter data with the help of twint and analyze some relationships based on followings and mentionings among a group of Twitter users.
We need twint library for scraping data, pandas for creating dataframes, and collections to get the grouped value counts in a list.
import twintimport pandas as pdfrom collections import Counter
Then we start with creating a user list that consists of twitter accounts. Our analysis will include the relationships of these users. I do not recommend to add users with more than 5K followings to this list by reason of the long code running time. Similarly, a long list might end up with the same problem as well.
I have chosen some popular Twitter users who everyone knows to make our analysis more interesting.
users = [    'shakira',    'KimKardashian',    'rihanna',    'jtimberlake',    'KingJames',    'neymarjr',]
Let’s start with relationship analysis and to use for this purpose write a function named get_followings that sends a request to twint library with a username. This function will return a list of users who our input user follows.
def get_followings(username):    c = twint.Config()    c.Username = username    c.Pandas = True    twint.run.Following(c)    list_of_followings = twint.storage.panda.Follow_df    return list_of_followings['following'][username]
Using get_followings function, we will get different following lists for every person in our users list and store the results to a dictionary (followings) and a list (following_list). following_list is a joined version of all followings and we will use it in order to calculate the most followed Twitter accounts in the next section.
The for loop below creates these two variables. Sometimes Twitter does not respond to our request and in this case, we get an Index Error. For such cases, I added an exception to the code to skip these users.
followings = {}following_list = []for person in users:    print('#####\nStarting: ' + person + '\n#####')    try:        followings[person] = get_followings(person)        following_list = following_list + followings[person]    except KeyError:        print('IndexError')
After getting all the following lists, we can simply calculate the most common values in the following_list variable to get the most popular accounts among our users. To get the most followed 10 accounts, we will use Counter function from collections library.
Counter(following_list).most_common(10)
The result of this function is shown below. Rihanna seems to be followed by all others and in our user group, she is definitely the most popular one.
[('rihanna', 5), ('RyanSeacrest', 3), ('BarackObama', 3), ('Oprah', 3), ('TheEllenShow', 3), ('kendricklamar', 3), ('KDTrey5', 3), ('Drake', 3), ('Nike', 3), ('NBA', 3)]
What if we want to see who is following who in our user group? To investigate it, I wrote a for loop that checks if anyone in the users is in the following list of another person. As a result, it creates a dictionary of lists showing the following statuses represented by Trues and Falses.
follow_relations ={}for following_user in followings.keys():    follow_relation_list = []    for followed_user in followings.keys():        if followed_user in followings[following_user]:            follow_relation_list.append(True)        else:            follow_relation_list.append(False)    follow_relations[following_user] = follow_relation_list
In the code below, the result dictionary is transformed into a pandas dataframe for a more user-friendly visualization. The rows of the dataframe show the users who are following, whereas the columns indicate the users who are followed.
following_df = pd.DataFrame.from_dict(follow_relations, orient='index', columns=followings.keys())
You can see the output of the analysis below. We confirm the popularity of Rihanna in this table again. She is followed by all others. However, for Kim Kardashian, we cannot speak in a similar way, according to the analysis, only Justin Timberlake in our user group follows her.
Mention counts are another strong relationship indicator between Twitter users. The function below (get_mention_count) is written for this purpose and it returns the mention counts between two users in one direction. We should put the mentioned username to the mention_word and in the function, an ‘@’ character is added to the beginning of it in order to separate mentions more precisely.
def get_mention_count(user, mention_word):    c = twint.Config()    c.Username = user    c.Search = '@' + mention_word    c.Store_object = True    twint.run.Search(c)    tweets = twint.output.tweets_list    mention_count = len(tweets)    tweets.clear()    return mention_count
In the analysis, we will use two nested for loops to retrieve mention counts of every user to all others in our group. As a result, we will get mention_relationships dictionary.
mention_relations = {}for mentioning_user in users:    print('#####\nStarting: ' + mentioning_user + '\n#####')    mention_count_list = []    for mentioned_user in users:        mention_count = get_mention_count(mentioning_user, mentioned_user)        mention_count_list.append(mention_count)    mention_relations[mentioning_user] = mention_count_list
Finally, we convert this dictionary to a pandas dataframe and it becomes an understandable and more easily interpretable table.
mention_df = pd.DataFrame.from_dict(mention_relations, orient='index', columns=mention_relations.keys())
And we see the output of the mention counts table below. Again, rows are showing the mentioning users and columns are indicating mentioned ones. The diagonal values are showing how many times users mentioned themselves and these are caused by retweets. If we ignore these values, we see that Lebron James is mentioned by everyone in the group and Rihanna looks like mentioned by everyone except Neymar. On the other side, no one in the group has ever mentioned Neymar in their tweets. Another interesting inference might be that Shakira mentioned Rihanna 52 times in her tweets however, Rihanna mentioned her only 7 times.
Conclusion
I tried to explain some basic social media analyses on famous Twitter users just for fun and meanwhile aimed to prepare them with the help of uncomplicated python codes. I hope you find them helpful. Lastly, it is certain that these analyses are open to improvement and if you have any advice or addition to the article, please do not hesitate to share it.