Twitter Trends Analysis using Python | by Saiteja Kura | Towards Data Science

Twitter, launched in 2006 is one of the most popular social media platforms available today. It helps by providing insights into popular trends and important cultural and political moments. In the Data Science industry twitter analysis can be used for tasks like marketing or product analysis. Twitter data has been used to analyze political polarisation and the spread of protest movements. In this article, we will learn the process of collecting twitter data, processing twitter text, and mapping twitter data geographically. We will be dealing with a subset of data having keywords #python and #javascript.
While collecting data, we are limited in two ways.1. Cannot collect data from the past(a year ago, etc.)2. Twitter only provides a sample of its data for free(like 1% of its data).
However, a 1% sample of data is in order of a few million tweets a day. Many Social Media companies have APIs which are made available to third party developers and researchers. Twitter has many API’s available.
According to tweepy docs, the Twitter streaming API is used to download twitter messages in real-time. It is useful for obtaining a high volume of tweets, or for creating a live feed using a site stream or user stream. It has two endpoints filter and sample. With a filter endpoint, users can request data using a few hundred keywords, a few thousand usernames, and 25 location ranges. With a sample endpoint, twitter will return 1 % of all twitter data. To collect data from Streaming API we will be using a package called tweepy which abstracts the work of setting up a stable Streaming API connection. We need to have our own twitter account and API keys for authentication.
tweepy requires an object called SListener which tells it how to handle incoming data. This SListener object opens a new timestamp file to store tweets and takes an optional API Argument. The below code will run until explicitly stopped(More the execution time, more the number of Twitter JSON objects in our file). The dataset in this article consists of 685 tweets(Twitter JSON objects) which were fetched from 7:39 pm IST to 7:44 pm IST.
Now a file named tweets.json will contain the JSON objects. The number of JSON objects depends on the opening and closing of the above connection.
json data format is a special data format that is human-readable and is easily transferred between machines. It is a combination of dictionaries and lists. Each JSON object has many child objects. In our case the main JSON object describes the main tweet, favourites_count, retweet_count, etc. and has nested dictionaries that describe the user, location, etc.
To analyze tweets at scale, it is better to store these tweets in a Pandas DataFrame. This allows us to apply analysis methods across rows and columns. However, with nested dictionaries the JSON object is complex. To overcome this problem, we will flatten the JSON object(maintain all attributes at one level instead of nesting).
Let us load the tweets list into a DataFrame.
df_tweet = pd.DataFrame(tweets)
Now let us compare the number of tweets having #python and #javascript. Let us write a function that will check for the given hashtag in all the text columns and return a series of boolean values indicating whether each row has that keyword or not.
We can notice that #python holds a slight lead over #javascript. Now, let us try and understand how mentions of the above two keywords change over time. Tweets about products, companies vary a lot wrt time. Let us try and capture the variation over time. When data is tagged with date and time it is known as Time Series data.
Firstly let us convert the created_at column to type DateTime. We need to create a metric that can be graphed over time. Let us create two columns python and js which consists of boolean values.
Now let us create a per minute average of mentions of both the hashtags and plot them across time. We will be using a Series method resample() that will allow us to summarize over a time window of our choice and apply an aggregate function to it.
resample(“1 min”).mean() will group the data minute wise and apply mean function to the grouped data.
As I already mentioned we fetched data from 7:39 pm to 7:44 pm we can notice that the minute's axis has values from 39 to 44 and we can notice how the frequency varies wrt time.
Sentiment analysis is a method of deriving meaning from text. It is a type of natural language processing method which determines whether a word, sentence, paragraph, document is positive or negative. Each document is given a positive or negative score based on the number of positive and negative words. This can be used to analyze the reactions over a product, company, etc.
We will use the VADER SentimentIntensityAnalyzer included in the Natural Language Toolkit or nltk. It is useful for analyzing short documents especially tweets. Considers emojis and capitalization of words too.
Each Sentiment score from the VADER Analyzer provides 4 values. Negative, Positive, Neutral, and Compound. The first 3 are self-explanatory and range between 0 and 1 whereas the compound value ranges from-1 to 1. Closer to 1 indicates positive and -1 indicates negative. We will again resample data minute-wise and find sentiment scores for every minute.
We can notice that, in the small subset of data we have, there are no negative tweets whereas javascript has more positive tweets in comparison to python.
We can sample data day-wise, month-wise depending on the data we collect and perform similar analyses to generate more insights.
The full code can be found here.
In this article, we learned how to fetch information from an API and we also learned how to deal with JSON data and perform time-series analysis on it. Much emphasis was not put on the internal explanation of Sentiment Analysis which would be covered in my future articles. As a part of my upcoming network analysis series, I will illustrate the network analysis of Twitter data using graph data structure.
Thanks for reading and feel free to share feedback! Stay home and stay safe.