Mining Twitter Data. Inside Social Media | by Paul Charles Owe | Towards Data Science

Twitter is a microblogging and social networking service where users post content and interact with posts known as “tweets”. Today, Twitter has popularized the use of hashtags as a way to group conversations and allow users to follow conversations on particular topics.
With over 500 million tweets per day, you can imagine how rich with information this platform is. The objective of this project is to gather and analyze twitter data in order to discover interesting information and hidden patterns.
This post serves as a “technical reflection” of my experience completing this task. I will be showing and briefly explaining relevant code snippets as well as results of my analysis. Here’s an overview of key steps I think are worth talking about.
AuthenticationData CollectionData Cleaning and PreprocessingModelling and AnalysisConclusion
Authentication
Data Collection
Data Cleaning and Preprocessing
Modelling and Analysis
Conclusion
In this application the authentication step was performed using the industry standard OAuth process which involves the user, consumer (the app) and the resource provider (Twitter). The key takeaway from this is that the exchange of credentials like username and password only happens between the user and the resource provider. All other exchanges are driven by tokens.
Creating a Twitter Developer Account. You need a developer account to have access to the twitter API. Once you create an application you may generate the Consumer Key, Consumer Secret, Access token and Access secret
I chose tweepy as my Twitter API client and I chose to create environment variables for my tokens as it promotes security and is language agnostic. Here is how I setup my twitter client.
import osimport sysfrom tweepy import APIfrom tweepy import OAuthHandlerdef get_twitter_auth():    #set up twitter authentication# Return: tweepy.OAuthHandler objecttry:        consumer_key = os.environ['TWITTER_CONSUMER_KEY']        consumer_secret = os.environ['TWITTER_CONSUMER_SECRET']        access_token = os.environ['TWITTER_ACCESS_TOKEN']        access_secret= os.environ['TWITTER_ACCESS_SECRET']    except KeyError:        sys.stderr.write("TWITTER_* environment variables not set\n")        sys.exit(1)    auth = OAuthHandler(consumer_key, consumer_secret)    auth.set_access_token(access_token, access_secret)    return authdef get_twitter_client():    #Setup twitter API client.# Return tweepy.API objectauth = get_twitter_auth()    client = API(auth)    return client
After the preliminary set ups. I could immediately start gathering data and interacting with the API.
Interacting with twitter’s REST API
All REST endpoints allow you to “go back in time”, meaning we are able to search for already published tweets.
Interacting with twitter’s Streaming API
The Streaming API looks into the future. Once you open a connection, you may leave it open and go forward in time. By keeping the HTTP connection open, we can retrieve all the tweets that match our filter criteria as they are published.
This is generally the preferred way of downloading a huge number of tweets.
In my case, I wanted to investigate realtime conversations among users during the US-Iran tension in December 2019 so I decided to stream all tweets with trending hashtags such as #USIran #USvsIran #Iranattacks
Here is how I implemented a custom stream listener.
class CustomListener(StreamListener):    """ Custom StreamListener for streaming twitter data."""def __init__(self, fname):        safe_fname = format_filename(fname)        self.outfile = "stream_%s.jsonl" % safe_fnamedef on_data(self, data):        #called when data is coming through. This method simply stores data as it is received in a .jsonl file. Each line in this file will contain a single tweet in json format        try:            with open(self.outfile, 'a') as f:                f.write(data)                return True        except BaseException as e:            sys.stderr.write("Error on_data: {}\n".format(e))            time.sleep(5)        return True                                ...if __name__ == '__main__':    query = sys.argv[1:] # list of CLI argumentsquery_fname    query_fname = ' '.join(query) # string    auth = get_twitter_auth()    twitter_stream = Stream(auth, CustomListener(query_fname))    twitter_stream.filter(track=query, is_async=True)
So, after streaming for about an hour, I managed to gather 13, 908 tweets from users who were talking about the US-Iran tension. I stored these tweets in a .jsonl file and it was time to pre-process the text. Here are some preprocessing tasks I performed:
Tokenization using nltk library
Tokenization refers to breaking down a stream of texts into tokens such as words, phrases, symbols. The content of a tweet includes emojis, user mentions, hasthags, urls etc. For this reason, when using the nltk library, I will showcase the use of the TweetTokenizer class as a tool to appropriately tokenize twitter content.
Stop word removal
Stop words are not content bearing and so we need to remove them. This category of words include articles, adverbs, symbols, punctuation, etc. Frequency analysis shows that these strings are the most frequently occurring in a document.
Normalization
Normalization is used to aggregate different terms in the same unit. Performing case normalization will help in automatically matching these strings with originally different casing so that they can be aggregated under the same term.
import sysimport stringimport jsonfrom collections import Counterfrom nltk.tokenize import TweetTokenizerfrom nltk.corpus import stopwordsimport matplotlib.pyplot as pltdef process(text, tokenizer=TweetTokenizer(), stopwords=[]):    """ Process tweet text:        - Lowercase        - tokenize        - stopword removal        - digits removal        Return: list of strings    """text = text.lower()    tokens = tokenizer.tokenize(text)    return [tok for tok in tokens if tok not in stopwords and not tok.isdigit()]if __name__ == '__main__':    fname = sys.argv[1]    tweet_tokenizer = TweetTokenizer()    punct = list(string.punctuation)    stopword_list = stopwords.words('english') + punct + ['rt', 'via', '...']tf = Counter()    with open(fname, 'r') as f:        for line in f:            tweet = json.loads(line)            tokens = process(text=tweet['text'], tokenizer=tweet_tokenizer, stopwords=stopword_list)            tf.update(tokens)        for tag, count in tf.most_common(20):print("{}: {}".format(tag, count))#plot resultsy = [count for tag, count in tf.most_common(20)]    x = range(1, len(y)+1)plt.bar(x, y)    plt.title("Term frequencies used in US-Iran Stream Data")    plt.ylabel("Frequency")    plt.savefig('us-iran-term-distn.png')
In the above snippet, I began the analysis by creating a visual representation of the term distribution of the 20 most common terms used in the tweets I collected.
Continuing the analysis, I searched the dataset for the key people everybody was talking about by finding their mention frequency.
With this information, I knew exactly which profiles I could start investigating to receive a more related feed. It was shocking, however, to discover that realdonaldtrump ranked 8th in popular user mentions related to this issue.
Time Series Analysis
A time series is a sequence of data points that consists of successive observation over a given time interval
It can be used to reorder tweets and track how users react to realtime events (such as — Sports, political elections, news)
In this application, I performed a time series analysis to monitor how users were tweeting as more news about this topic was being released. It seems to me as though within the hour I was streaming, users were actively tweeting about the topic throughout, with a peak of 200 tweets collected around 10:45AM
Users, Followers and Friends
I decided to explore some of the top users’ profiles, list of followers and “following” (or friends) for further analysis. Here is how I pulled this information from twitter.
#get followers for a given userfname = "Twitter_Profiles/{}/followers.jsonl".format(screen_name)    with open(fname, 'w') as f:        for followers in Cursor(client.followers_ids, screen_name=screen_name).pages(max_pages):            for chunk in paginate(followers, 100):                users = client.lookup_users(user_ids=chunk)                for user in users:                    f.write(json.dumps(user._json)+"\n")            if len(followers) == 5000:                print("More results available. Sleeping for 60 seconds to avoid rate limit")                time.sleep(60)#get friends for a given userfname = "Twitter_Profiles/{}/friends.jsonl".format(screen_name)    with open(fname, 'w') as f:        for friends in Cursor(client.friends_ids, screen_name=screen_name).pages(max_pages):            for chunk in paginate(friends, 100):                users = client.lookup_users(user_ids=chunk)                for user in users:                    f.write(json.dumps(user._json)+"\n")            if len(friends) == 5000:                print("More results available. Sleeping for 60 seconds to avoid rate limit")                time.sleep(60)# get user's profile    fname = "Twitter_Profiles/{}/user_profile.json".format(screen_name)    with open(fname, 'w') as f:        profile = client.get_user(screen_name=screen_name)        f.write(json.dumps(profile._json, indent=4))
Measuring Influence and Engagement
A metric of engagement provides an assessment of users’ responses to tweets and other content, often created with the purpose to drive traffic. On twitter, users engage by means of liking or retweeting — which provides more visibility to the tweets
Understanding the influence of a user involves analyzing the user’s reach in combination with their tweet engagement statistics.
#Build up a list of followers    followers_file1 = 'Twitter_Profiles/{}/followers.jsonl'.format(screen_name1)    followers_file2 = 'Twitter_Profiles/{}/followers.jsonl'.format(screen_name2)with open(followers_file1) as f1, open(followers_file2) as f2:        reach1 = []        reach2 = []        for line in f1:            profile = json.loads(line)            reach1.append((profile['screen_name'], profile['followers_count']))        for line in f2:            profile = json.loads(line)            reach2.append((profile['screen_name'], profile['followers_count']))#Load basic statistics    profile_file1 = 'Twitter_Profiles/{}/user_profile.json'.format(screen_name1)    profile_file2 = 'Twitter_Profiles/{}/user_profile.json'.format(screen_name2)    with open(profile_file1) as f1, open(profile_file2) as f2:        profile1 = json.load(f1)        profile2 = json.load(f2)followers1 = profile1['followers_count']        followers2 = profile2['followers_count']tweets1 = profile['statuses_count']        tweets2 = profile['statuses_count']sum_reach1 = sum([x[1] for x in reach1]) #sum up all of a user's followers, followers    sum_reach2 = sum([x[1] for x in reach2])    avg_followers1 = round(sum_reach1/ followers1, 2)    avg_followers2 = round(sum_reach2/ followers2, 2)#Load the timelines for two users to observe the number of times their tweets have been favorited    timeline_file1 = 'user_timeline_{}.jsonl'.format(screen_name1)    timeline_file2 = 'user_timeline_{}.jsonl'.format(screen_name2)    with open(timeline_file1) as f1, open(timeline_file2) as f2:        favorited_count1, retweet_count1 = [], []        favorited_count2, retweet_count2 = [], []        for line in f1:            tweet = json.loads(line)            favorited_count1.append(tweet['favorite_count'])            retweet_count1.append(tweet['retweet_count'])for line in f2:            tweet = json.loads(line)            favorited_count2.append(tweet['favorite_count'])            retweet_count2.append(tweet['retweet_count'])
Complete scripts for doing this can be found on my GitHub
Mining Social Communities
A social community is a group of people who share some common conditions e.g geographical area, same religious beliefs, or people who share same interests.
Explicit community — people with common interests and explicitly know exactly whether they belong to a community or not and typically understand who other members in the said community are.
Implicit communities — come into existence when members share common interests without having clear and strong connections between each other
In the next section, I will demonstrate how I analyzed user data in order to segment a bunch of user profiles into groups with the purpose of uncovering a user’s Implicit Communities.
Cluster analysis
Cluster Analysis is a machine learning technique used to group items in such a way that objects in the same cluster are similar to each other and dissimilar to objects in other clusters. Here are some of the key takeaways from the script.
I used the K-means algorithm to achieve this purpose.
Each user profile is represented as a vector. Based on user description, I used the TF-IDF approach to measure the weight/ importance of a word in a given context
We can limit the number of features being extracted by the TfidfVectorizer (by explicitly specifying max_features or specifying range of min_df and max_df). The idea of excluding features based on document frequency is to avoid features that are not representative. Limiting them can also be used to speed up the computation.
An ngram is a contiguous sequence of n items and in this context, it refers to text which is tokenized into a sequence of words. The benefit of using ngrams is its ability of capturing phrases.
Heres how I performed the cluster analysis based on user bios.
from sklearn.feature_extraction.text import TfidfVectorizerfrom sklearn.cluster import KMeansdef get_parser():    parser = ArgumentParser("Clustering for followers")    parser.add_argument('--filename')     parser.add_argument('--k', type=int)     parser.add_argument('--min-df', type=int, default=2)     parser.add_argument('--max-df', type=float, default=0.8)     parser.add_argument('--max-features', type=int, default=None)     parser.add_argument('--no-idf', dest='user_idf', default=True, action='store_false')     parser.add_argument('--min-ngram', type=int, default=1)     parser.add_argument('--max-ngram', type=int, default=1)     return parserif __name__ == '__main__':    parser = get_parser()    args = parser.parse_args()    if args.min_ngram > args.max_ngram:        print("Error: incorrect value for --min--ngram ({}): it cant be higher than \        --max--value ({})".format(args.min_ngram, args.max_ngram))        sys.exit(1)    with open(args.filename) as f:        #load datausers = []        for line in f:            profile = json.loads(line)            users.append(profile['description'])        #create vectorizer        vectorizer = TfidfVectorizer(max_df=args.max_df,                                    min_df=args.min_df,                                    max_features=args.max_features,                                    stop_words='english',                                    ngram_range=(args.min_ngram, args.max_ngram),                                    use_idf=args.user_idf)#fit data        X = vectorizer.fit_transform(users)        print("Data dimensions: {}".format(X.shape))#perform clustering        km = KMeans(n_clusters=args.k)        km.fit(X)        clusters = defaultdict(list)        for i, label in enumerate(km.labels_):            clusters[label].append(users[i])#print 10 user description of this clusterfor label, description in clusters.items():            print("--------- Cluster {}".format(label+i))            for desc in description[:10]:                print(desc)
So you can tell that the script performed fairly well in segmenting users based on the bio they have on their profiles. I used my twitter profile for this example and you can tell that I have a couple of investors, tech enthusiasts, entrepreneurs and religious people in my community. This reminds me of a popular saying — “Show me your friends and I will tell you who you are”. The script could as well be used to delve deeper into other users’ profiles in order to uncover implicit communities they belong to.
Graph Mining and Analyzing Conversations
When many users are engaging in conversation, a “reply to” network emerges. The image below represents a conversation model on twitter with tweets as nodes, and edges as “reply to” relationship.
Naturally, this kind of structure can easily be mapped onto a Graph data structure. Some key takeaways from this model:
One-to-Many cardinality
No cycles
For these reasons I modelled a Directed Acyclic Graph and used properties of this graph and algorithms from graph theory to mine conversations. Here is the code snippet of the graph mining technique I used.
import sysimport jsonfrom operator import itemgetterimport networkx as nxdef usage():    print("Usage")    print("python {} <filename>".format(sys.argv[0]))if __name__ == "__main__":    if len(sys.argv) != 2:        usage()        sys.exit(1)fname = sys.argv[1]    with open(fname) as f: #takes in a jsonl file of tweets as input        graph = nx.DiGraph()        for line in f:            tweet = json.loads(line)            if 'id' in tweet:                graph.add_node(tweet['id'],                                tweet=tweet['text'],                                author=tweet['user']['screen_name'],                                created_at=tweet['created_at'])                if tweet['in_reply_to_status_id']:                    reply_to = tweet['in_reply_to_status_id']                    if reply_to in graph and tweet['user']['screen_name'] != graph.node[reply_to]['author']:  #if the user is not replying to themselves                        graph.add_edge(tweet['in_reply_to_status_id'], tweet['id'])        #Print some basic statsprint(nx.info(graph))#Find most replied tweet         sorted_replied = sorted(graph.degree(), key=itemgetter(1), reverse=True)most_replied_id, replies = sorted_replied[0]        print("Most replied tweet ({} replies:".format(replies))        print(graph.node[most_replied_id])#Find longest conversation (longest path)        print("Longest discussion:")        longest_path = nx.dag_longest_path(graph)        for tweet_id in longest_path:            node = graph.node[tweet_id]            print("{} (by {} at {})".format(node['tweet'], node['author'], node['created_at']))
In-degree of a node: number of replies of a given tweet
Root: the start of a conversation
Leaf: the end of a conversation
Longest Path Algorithm: Used to find the tweet with the longest conversation.
I used networkX library because it provides efficient computation of graph structures
Dynamic Maps
Finally, to conclude my analysis, I created a visual representation of where these tweets are coming from by plotting them on a map.
Extraction of geographical data
Using GeoJSON library I extracted geometric data for each tweet from a stream file. I created an output file with a geo.json extension of this geographical data
#read dataset of tweets in jsonl file and produce a geoson file associted with geographical dataimport jsonfrom argparse import ArgumentParserdef get_parser():    parser = ArgumentParser()    parser.add_argument('--tweets')    parser.add_argument('--geojson')    return parserif __name__ == "__main__":    parser = get_parser()    args = parser.parse_args()#Read tweet collection and build geo data structure.    with open(args.tweets, 'r') as f: #read dataset of tweets        geo_data = {        'type' : "FeatureCollection",        'features' : []        }for line in f:            tweet = json.loads(line)            try:                if tweet['coordinates']:                    geo_json_feature = {                        "type" : "Feature",                        "geometry": {                        "type" : "Point",                        "coordinates" : tweet['coordinates']['coordinates']                        },                        "properties": {                        "text" : tweet['text'],                        "created_at" : tweet['created_at']                        }                    }                    geo_data['features'].append(geo_json_feature)            except KeyError:                #Skip if json doc is not a tweet (errors, etc)                continue    #Save geo data    with open(args.geojson, 'w') as fout:        fout.write(json.dumps(geo_data, indent=4))
Creating Dynamic Maps
I used the Folium library to generate an interactive map. The advantage is that Folium handles the conversion between python data structures and Javascript, HTML and CSS components seamlessly.
from argparse import ArgumentParserimport foliumfrom folium.plugins import MarkerClusterimport jsondef get_parser():    parser = ArgumentParser()    parser.add_argument('--geojson')    parser.add_argument('--map')    return parserdef make_map(geojson_file, map_file):tweet_map = folium.Map(Location=[50, 5], max_zoom=20)marker_cluster = MarkerCluster().add_to(tweet_map)geodata= json.load(open(geojson_file))for tweet in geodata['features']:        tweet['geometry']['coordinates'].reverse()        marker = folium.Marker(tweet['geometry']['coordinates'], popup=tweet['properties']['text'])        marker.add_to(marker_cluster)#Save to HTML map file    tweet_map.save(map_file)if __name__ == '__main__':    parser = get_parser()    args = parser.parse_args()    make_map(args.geojson, args.map)
As you can see, this script visualizes the different locations from which people were tweeting about the death of NBA legend Kobe Bryant when it happened. The script clusters users tweeting from a particular area and precisely locates the exact point where each tweet was posted.
These results are generated from a 15-minute stream of the storm of tweets a few hours after the event was publicized. You may view and interact with the full map here
This was all for Mining Twitter Data. I really enjoyed working on this project. I have full source code available on my GitHub for anyone interested in implementing this project.