Mining Twitter Data
By: Paul Charles Owe
Tags: Data Analytics, First Post, Twitter, Data Mining, Data Science

Mining Twitter Data

Photo by Lauren Feiner from cnbc.com

Twitter is a microblogging and social networking service where users post content and interact with posts known as “tweets”. Today, Twitter has popularized the use of hashtags as a way to group conversations and allow users to follow conversations on particular topics.

Objective

With over 500 million tweets per day, you can imagine how rich with information this platform is. The objective of this project is to gather and analyze twitter data in order to discover interesting information and hidden patterns.

This post serves as a “technical reflection” of my experience completing this task. I will be showing and briefly explaining relevant code snippets as well as results of my analysis. Here’s an overview of key steps I think are worth talking about.

Overview

Authentication Data Collection Data Cleaning and Preprocessing Modelling and Analysis Conclusion

Authentication

In this application the authentication step was performed using the industry standard OAuth process which involves the user, consumer (the app) and the resource provider (Twitter). The key takeaway from this is that the exchange of credentials like username and password only happens between the user and the resource provider. All other exchanges are driven by tokens.

Photo of the OAuth Procedure by Oracle

Creating a Twitter Developer Account. You need a developer account to have access to the twitter API. Once you create an application you may generate the Consumer Key, Consumer Secret, Access token and Access secret

I chose tweepy as my Twitter API client and I chose to create environment variables for my tokens as it promotes security and is language agnostic. Here is how I setup my twitter client.

import os

import sys

from tweepy import API

from tweepy import OAuthHandler def get_twitter_auth():

#set up twitter authentication # Return: tweepy.OAuthHandler object try:

consumer_key = os.environ['TWITTER_CONSUMER_KEY']

consumer_secret = os.environ['TWITTER_CONSUMER_SECRET']

access_token = os.environ['TWITTER_ACCESS_TOKEN']

access_secret= os.environ['TWITTER_ACCESS_SECRET']

except KeyError:

sys.stderr.write("TWITTER_* environment variables not set

")

sys.exit(1)

auth = OAuthHandler(consumer_key, consumer_secret)

auth.set_access_token(access_token, access_secret)

return auth def get_twitter_client():

#Setup twitter API client. # Return tweepy.API object auth = get_twitter_auth()

client = API(auth)

return client

Data Collection

After the preliminary set ups. I could immediately start gathering data and interacting with the API.

Interacting with twitter’s REST API

All REST endpoints allow you to “go back in time”, meaning we are able to search for already published tweets.

Interacting with twitter’s Streaming API

The Streaming API looks into the future. Once you open a connection, you may leave it open and go forward in time. By keeping the HTTP connection open, we can retrieve all the tweets that match our filter criteria as they are published.

This is generally the preferred way of downloading a huge number of tweets.

In my case, I wanted to investigate realtime conversations among users during the US-Iran tension in December 2019 so I decided to stream all tweets with trending hashtags such as #USIran #USvsIran #Iranattacks

Here is how I implemented a custom stream listener.

class CustomListener(StreamListener):

""" Custom StreamListener for streaming twitter data.""" def __init__(self, fname):

safe_fname = format_filename(fname)

self.outfile = "stream_%s.jsonl" % safe_fname def on_data(self, data): #called when data is coming through. This method simply stores data as it is received in a .jsonl file. Each line in this file will contain a single tweet in json format try:

with open(self.outfile, 'a') as f:

f.write(data)

return True

except BaseException as e:

sys.stderr.write("Error on_data: {}

".format(e))

time.sleep(5)

return True

... if __name__ == '__main__':

query = sys.argv[1:] # list of CLI argumentsquery_fname

query_fname = ' '.join(query) # string

auth = get_twitter_auth()

twitter_stream = Stream(auth, CustomListener(query_fname))

twitter_stream.filter(track=query, is_async=True)

Data Cleaning and Preprocessing

So, after streaming for about an hour, I managed to gather 13, 908 tweets from users who were talking about the US-Iran tension. I stored these tweets in a .jsonl file and it was time to pre-process the text. Here are some preprocessing tasks I performed:

Tokenization using nltk library

Tokenization refers to breaking down a stream of texts into tokens such as words, phrases, symbols. The content of a tweet includes emojis, user mentions, hasthags, urls etc. For this reason, when using the nltk library, I will showcase the use of the TweetTokenizer class as a tool to appropriately tokenize twitter content.

Stop word removal

Stop words are not content bearing and so we need to remove them. This category of words include articles, adverbs, symbols, punctuation, etc. Frequency analysis shows that these strings are the most frequently occurring in a document.

Normalization

Normalization is used to aggregate different terms in the same unit. Performing case normalization will help in automatically matching these strings with originally different casing so that they can be aggregated under the same term.

import sys

import string

import json

from collections import Counter

from nltk.tokenize import TweetTokenizer

from nltk.corpus import stopwords

import matplotlib.pyplot as plt def process(text, tokenizer=TweetTokenizer(), stopwords=[]):

""" Process tweet text:

- Lowercase

- tokenize

- stopword removal

- digits removal

Return: list of strings

""" text = text.lower()

tokens = tokenizer.tokenize(text)

return [tok for tok in tokens if tok not in stopwords and not tok.isdigit()] if __name__ == '__main__':

fname = sys.argv[1]

tweet_tokenizer = TweetTokenizer()

punct = list(string.punctuation)

stopword_list = stopwords.words('english') + punct + ['rt', 'via', '...'] tf = Counter()

with open(fname, 'r') as f:

for line in f:

tweet = json.loads(line)

tokens = process(text=tweet['text'], tokenizer=tweet_tokenizer, stopwords=stopword_list)

tf.update(tokens)

for tag, count in tf.most_common(20): print("{}: {}".format(tag, count)) #plot results y = [count for tag, count in tf.most_common(20)]

x = range(1, len(y)+1) plt.bar(x, y)

plt.title("Term frequencies used in US-Iran Stream Data")

plt.ylabel("Frequency")

plt.savefig('us-iran-term-distn.png')

Modelling and Analysis

In the above snippet, I began the analysis by creating a visual representation of the term distribution of the 20 most common terms used in the tweets I collected.

Top 20 used terms and the term distribution

Continuing the analysis, I searched the dataset for the key people everybody was talking about by finding their mention frequency.

Mention frequency results

With this information, I knew exactly which profiles I could start investigating to receive a more related feed. It was shocking, however, to discover that realdonaldtrump ranked 8th in popular user mentions related to this issue.

Time Series Analysis

A time series is a sequence of data points that consists of successive observation over a given time interval

It can be used to reorder tweets and track how users react to realtime events (such as — Sports, political elections, news)

In this application, I performed a time series analysis to monitor how users were tweeting as more news about this topic was being released. It seems to me as though within the hour I was streaming, users were actively tweeting about the topic throughout, with a peak of 200 tweets collected around 10:45AM

Users, Followers and Friends

I decided to explore some of the top users’ profiles, list of followers and “following” (or friends) for further analysis. Here is how I pulled this information from twitter.

#get followers for a given user fname = "Twitter_Profiles/{}/followers.jsonl".format(screen_name)

with open(fname, 'w') as f:

for followers in Cursor(client.followers_ids, screen_name=screen_name).pages(max_pages):

for chunk in paginate(followers, 100):

users = client.lookup_users(user_ids=chunk)

for user in users:

f.write(json.dumps(user._json)+"

")

if len(followers) == 5000:

print("More results available. Sleeping for 60 seconds to avoid rate limit")

time.sleep(60) #get friends for a given user fname = "Twitter_Profiles/{}/friends.jsonl".format(screen_name)

with open(fname, 'w') as f:

for friends in Cursor(client.friends_ids, screen_name=screen_name).pages(max_pages):

for chunk in paginate(friends, 100):

users = client.lookup_users(user_ids=chunk)

for user in users:

f.write(json.dumps(user._json)+"

")

if len(friends) == 5000:

print("More results available. Sleeping for 60 seconds to avoid rate limit")

time.sleep(60) # get user's profile

fname = "Twitter_Profiles/{}/user_profile.json".format(screen_name)

with open(fname, 'w') as f:

profile = client.get_user(screen_name=screen_name)

f.write(json.dumps(profile._json, indent=4))

Measuring Influence and Engagement

A metric of engagement provides an assessment of users’ responses to tweets and other content, often created with the purpose to drive traffic. On twitter, users engage by means of liking or retweeting — which provides more visibility to the tweets

Understanding the influence of a user involves analyzing the user’s reach in combination with their tweet engagement statistics.

#Build up a list of followers

followers_file1 = 'Twitter_Profiles/{}/followers.jsonl'.format(screen_name1)

followers_file2 = 'Twitter_Profiles/{}/followers.jsonl'.format(screen_name2) with open(followers_file1) as f1, open(followers_file2) as f2:

reach1 = []

reach2 = []

for line in f1:

profile = json.loads(line)

reach1.append((profile['screen_name'], profile['followers_count']))

for line in f2:

profile = json.loads(line)

reach2.append((profile['screen_name'], profile['followers_count'])) #Load basic statistics

profile_file1 = 'Twitter_Profiles/{}/user_profile.json'.format(screen_name1)

profile_file2 = 'Twitter_Profiles/{}/user_profile.json'.format(screen_name2)

with open(profile_file1) as f1, open(profile_file2) as f2:

profile1 = json.load(f1)

profile2 = json.load(f2) followers1 = profile1['followers_count']

followers2 = profile2['followers_count'] tweets1 = profile['statuses_count']

tweets2 = profile['statuses_count'] sum_reach1 = sum([x[1] for x in reach1]) #sum up all of a user's followers, followers

sum_reach2 = sum([x[1] for x in reach2])

avg_followers1 = round(sum_reach1/ followers1, 2)

avg_followers2 = round(sum_reach2/ followers2, 2) #Load the timelines for two users to observe the number of times their tweets have been favorited

timeline_file1 = 'user_timeline_{}.jsonl'.format(screen_name1)

timeline_file2 = 'user_timeline_{}.jsonl'.format(screen_name2)

with open(timeline_file1) as f1, open(timeline_file2) as f2:

favorited_count1, retweet_count1 = [], []

favorited_count2, retweet_count2 = [], []

for line in f1:

tweet = json.loads(line)

favorited_count1.append(tweet['favorite_count'])

retweet_count1.append(tweet['retweet_count']) for line in f2:

tweet = json.loads(line)

favorited_count2.append(tweet['favorite_count'])

retweet_count2.append(tweet['retweet_count'])

Complete scripts for doing this can be found on my GitHub

Mining Social Communities

A social community is a group of people who share some common conditions e.g geographical area, same religious beliefs, or people who share same interests.

Explicit community — people with common interests and explicitly know exactly whether they belong to a community or not and typically understand who other members in the said community are.

— people with common interests and explicitly know exactly whether they belong to a community or not and typically understand who other members in the said community are. Implicit communities — come into existence when members share common interests without having clear and strong connections between each other

In the next section, I will demonstrate how I analyzed user data in order to segment a bunch of user profiles into groups with the purpose of uncovering a user’s Implicit Communities.

Cluster analysis

Cluster Analysis is a machine learning technique used to group items in such a way that objects in the same cluster are similar to each other and dissimilar to objects in other clusters. Here are some of the key takeaways from the script.

I used the K-means algorithm to achieve this purpose.

Each user profile is represented as a vector. Based on user description, I used the TF-IDF approach to measure the weight/ importance of a word in a given context

We can limit the number of features being extracted by the TfidfVectorizer (by explicitly specifying max_features or specifying range of min_df and max_df). The idea of excluding features based on document frequency is to avoid features that are not representative. Limiting them can also be used to speed up the computation.

An ngram is a contiguous sequence of n items and in this context, it refers to text which is tokenized into a sequence of words. The benefit of using ngrams is its ability of capturing phrases.

Heres how I performed the cluster analysis based on user bios.

from sklearn.feature_extraction.text import TfidfVectorizer

from sklearn.cluster import KMeans def get_parser():

parser = ArgumentParser("Clustering for followers")

parser.add_argument('--filename')

parser.add_argument('--k', type=int)

parser.add_argument('--min-df', type=int, default=2)

parser.add_argument('--max-df', type=float, default=0.8)

parser.add_argument('--max-features', type=int, default=None)

parser.add_argument('--no-idf', dest='user_idf', default=True, action='store_false')

parser.add_argument('--min-ngram', type=int, default=1)

parser.add_argument('--max-ngram', type=int, default=1)

return parser if __name__ == '__main__':

parser = get_parser()

args = parser.parse_args()

if args.min_ngram > args.max_ngram:

print("Error: incorrect value for --min--ngram ({}): it cant be higher than \

--max--value ({})".format(args.min_ngram, args.max_ngram))

sys.exit(1)

with open(args.filename) as f:

#load data users = []

for line in f:

profile = json.loads(line)

users.append(profile['description'])

#create vectorizer

vectorizer = TfidfVectorizer(max_df=args.max_df,

min_df=args.min_df,

max_features=args.max_features,

stop_words='english',

ngram_range=(args.min_ngram, args.max_ngram),

use_idf=args.user_idf) #fit data

X = vectorizer.fit_transform(users)

print("Data dimensions: {}".format(X.shape)) #perform clustering

km = KMeans(n_clusters=args.k)

km.fit(X)

clusters = defaultdict(list)

for i, label in enumerate(km.labels_):

clusters[label].append(users[i]) #print 10 user description of this cluster for label, description in clusters.items():

print("--------- Cluster {}".format(label+i))

for desc in description[:10]:

print(desc)

Results from cluster analysis

So you can tell that the script performed fairly well in segmenting users based on the bio they have on their profiles. I used my twitter profile for this example and you can tell that I have a couple of investors, tech enthusiasts, entrepreneurs and religious people in my community. This reminds me of a popular saying — “Show me your friends and I will tell you who you are”. The script could as well be used to delve deeper into other users’ profiles in order to uncover implicit communities they belong to.

Graph Mining and Analyzing Conversations

When many users are engaging in conversation, a “reply to” network emerges. The image below represents a conversation model on twitter with tweets as nodes, and edges as “reply to” relationship.

Conversation Model

Naturally, this kind of structure can easily be mapped onto a Graph data structure. Some key takeaways from this model:

One-to-Many cardinality

No cycles

For these reasons I modelled a Directed Acyclic Graph and used properties of this graph and algorithms from graph theory to mine conversations. Here is the code snippet of the graph mining technique I used.

import sys

import json

from operator import itemgetter

import networkx as nx def usage():

print("Usage")

print("python {} <filename>".format(sys.argv[0])) if __name__ == "__main__":

if len(sys.argv) != 2:

usage()

sys.exit(1) fname = sys.argv[1]

with open(fname) as f: #takes in a jsonl file of tweets as input

graph = nx.DiGraph()

for line in f:

tweet = json.loads(line)

if 'id' in tweet:

graph.add_node(tweet['id'],

tweet=tweet['text'],

author=tweet['user']['screen_name'],

created_at=tweet['created_at'])

if tweet['in_reply_to_status_id']:

reply_to = tweet['in_reply_to_status_id']

if reply_to in graph and tweet['user']['screen_name'] != graph.node[reply_to]['author']: #if the user is not replying to themselves

graph.add_edge(tweet['in_reply_to_status_id'], tweet['id'])



#Print some basic stats print(nx.info(graph)) #Find most replied tweet

sorted_replied = sorted(graph.degree(), key=itemgetter(1), reverse=True) most_replied_id, replies = sorted_replied[0]

print("Most replied tweet ({} replies:".format(replies))

print(graph.node[most_replied_id]) #Find longest conversation (longest path)

print("Longest discussion:")

longest_path = nx.dag_longest_path(graph)

for tweet_id in longest_path:

node = graph.node[tweet_id]

print("{} (by {} at {})".format(node['tweet'], node['author'], node['created_at']))

In-degree of a node: number of replies of a given tweet

Root: the start of a conversation

Leaf: the end of a conversation

Longest Path Algorithm: Used to find the tweet with the longest conversation.

I used networkX library because it provides efficient computation of graph structures

Dynamic Maps

Finally, to conclude my analysis, I created a visual representation of where these tweets are coming from by plotting them on a map.

Extraction of geographical data

Using GeoJSON library I extracted geometric data for each tweet from a stream file. I created an output file with a geo.json extension of this geographical data

#read dataset of tweets in jsonl file and produce a geoson file associted with geographical data

import json

from argparse import ArgumentParser def get_parser():

parser = ArgumentParser()

parser.add_argument('--tweets')

parser.add_argument('--geojson')

return parser if __name__ == "__main__":

parser = get_parser()

args = parser.parse_args() #Read tweet collection and build geo data structure.

with open(args.tweets, 'r') as f: #read dataset of tweets

geo_data = {

'type' : "FeatureCollection",

'features' : []

} for line in f:

tweet = json.loads(line)

try:

if tweet['coordinates']:

geo_json_feature = {

"type" : "Feature",

"geometry": {

"type" : "Point",

"coordinates" : tweet['coordinates']['coordinates']

},

"properties": {

"text" : tweet['text'],

"created_at" : tweet['created_at']

}

}

geo_data['features'].append(geo_json_feature)

except KeyError:

#Skip if json doc is not a tweet (errors, etc)

continue

#Save geo data

with open(args.geojson, 'w') as fout:

fout.write(json.dumps(geo_data, indent=4))

Creating Dynamic Maps

I used the Folium library to generate an interactive map. The advantage is that Folium handles the conversion between python data structures and Javascript, HTML and CSS components seamlessly.

from argparse import ArgumentParser

import folium

from folium.plugins import MarkerCluster

import json def get_parser():

parser = ArgumentParser()

parser.add_argument('--geojson')

parser.add_argument('--map')

return parser def make_map(geojson_file, map_file): tweet_map = folium.Map(Location=[50, 5], max_zoom=20) marker_cluster = MarkerCluster().add_to(tweet_map) geodata= json.load(open(geojson_file)) for tweet in geodata['features']:

tweet['geometry']['coordinates'].reverse()

marker = folium.Marker(tweet['geometry']['coordinates'], popup=tweet['properties']['text'])

marker.add_to(marker_cluster) #Save to HTML map file

tweet_map.save(map_file) if __name__ == '__main__':

parser = get_parser()

args = parser.parse_args()

make_map(args.geojson, args.map)

Dynamic Map (zoomed out)

Dynamic Map (zoomed in)

As you can see, this script visualizes the different locations from which people were tweeting about the death of NBA legend Kobe Bryant when it happened. The script clusters users tweeting from a particular area and precisely locates the exact point where each tweet was posted.

These results are generated from a 15-minute stream of the storm of tweets a few hours after the event was publicized. You may view and interact with the full map here

CONCLUSION

This was all for Mining Twitter Data. I really enjoyed working on this project. I have full source code available on my GitHub for anyone interested in implementing this project.