Sentiment analysis on streaming Twitter data using Spark Structured Streaming & Python
By: Elena Stamatelou
Tags: Python, Sentiment Analysis, Spark, Streaming, Twitter

As a Data Scientist, in the era of continuous data generation, you need to process data in real-time. A traditional data processing system such as the Extract Transform Load (ETL) process is only adequate for the processing of static data. Traditional ETL tools process data in batches and are not capable of handling real-time data, but only data that is already stored in a database system.

For the processing of real-time data, we need systems that support stream processing, such as Apache Spark. Those processing systems provide the option for continuous computations, as data is continuously flowing through them. Examples of such functionalities are the regular cleaning and aggregation of the incoming data before storage.

Apache Spark provides two ways of working with streaming data; Spark Streaming and Spark Structured Streaming. In this tutorial, we use Spark Structure Streaming because is more inclined towards real-time stream processing. Read this article for a comparison between the two different ways of working.

Our goal

This tutorial is a good starting point for those who have little or no experience with Apache Spark Streaming. We use Twitter data since Twitter provides an API that is easy for developers to access.

With this tutorial, you can also make a sentiment analysis about a topic of your interest. You can choose to query tweets that contain a specific keyword. Just try out different keywords and make your analysis!

What are the real-time dominant sentiments of the public opinion regarding a specific topic or hashtag? Is the public positive or negative about it? Does the average person think subjectively or objectively?

We present an end-to-end architecture on how to stream data from Twitter, clean it, and apply a simple sentiment analysis model to detect the polarity and subjectivity of each tweet.

You can find the code used in this tutorial in our GitHub repository.

The Architecture

End-to-end architecture, Image by Author

Part 1: Send tweets from the Twitter API

In this part, we use our developer credentials to authenticate and connect to the Twitter API. We also create a TCP socket between Twitter’s API and Spark, which waits for the call of the Spark Structured Streaming and then sends the Twitter data. Here, we use Python’s Tweepy library for connecting and getting the tweets from the Twitter API.

Part 2: Tweet preprocessing and sentiment analysis

In this part, we receive the data from the TCP socket and preprocess it with the pyspark library, which is Python’s API for Spark. Then, we apply sentiment analysis using textblob, which is Python’s library for processing textual data. After sentiment analysis, we save the tweet and the sentiment analysis scores in a parquet file, which is a data storage format.

Let’s start with the tutorial!

Part 1: Send tweets from the Twitter API

Step 1: Import the necessary packages

The tweepy library is necessary for connecting to the Twitter API and building the data streaming pipeline. We import its classes; Stream and StreamListener for building the stream, and OAuthHandler for authenticating to Twitter. We import the socket module to create a communication channel between our local machine and the Twitter API. We need the JSON module to handle the data of JSON objects.

Step 2: Insert your credentials

Use your credentials from your Twitter developer account. If you do not have them yet, you will need to register on Twitter for a developer account and then request your credentials from the Development Portal.

Step 3: Create a StreamListener instance

The class TweetListener represents a StreamListener instance, which connects to the Twitter API and returns one tweet at a time. As soon as we activate the Stream, it continuously creates instances.

The class consists of 3 methods; the on_data , the on_error , and the __init__ .

The on_data method reads the incoming tweet JSON file, which contains one tweet, and defines which part to keep. Some example parts could be the actual tweet message, comments, or hashtags. In our case, we want to extract the actual text of the tweet. If we only request the ['text'] field from each tweet, we will only receive messages shorter than 140 characters. To be sure we receive the full message, we need to first check if the tweet is longer than 140 characters. If it is, we extract the ['extended_tweet']['full_text'] field and if it is not; we extract the ['text'] field. At the end of each tweet, we add the 't_end' string, so that at a later stage, we can identify the end of each tweet easier.

The on_error method makes sure that the stream works and the __init__ method initializes the socket of the Twitter API.

Step 4: Send data from Twitter

To get data from the Twitter API, we first use the pre-defined credentials to authenticate the connection to the API. After the authentication, we stream the tweet data objects that contain a selected keyword and language. The returned objects are tweets of the TweetListener class.

Step 5: Start Streaming

Before streaming the data from the Twitter API, we need to create a listening TCP socket in the local machine (server) in a pre-defined local IP address and a port. The socket consists of a server-side, which is our local machine, and a client-side, which will be the Twitter API. The open socket from the server-side listens for the client. When the client-side is up and running, the socket will receive the data from the Twitter API. To query the tweets that are related to a specific topic, we also select one or multiple keywords for streaming.

Part 2: Tweet preprocessing and sentiment analysis

Step 1: Import the necessary packages

We use pyspark , which is the Python API for Spark. Here, we use Spark Structured Streaming, which is a stream processing engine built on the Spark SQL engine and that’s why we import the pyspark.sql module. We import its classes; SparkSession to create a stream session, function, and types to make a list of built-in functions and data types available. We also use textblob for the tweet text classification.

Step 2: Tweet preprocessing

We preprocess the tweets so we can have only the clean text of the tweet. In each batch, we receive many tweets from the Twitter API and split the tweets at the string t_end . Then, we remove the empty rows and apply regular expressions to clean up the tweet text. In more detail; we remove the links (https://..) , the usernames (@..) , the hashtags (#), the string that shows if the current tweet is a retweet (RT) , and the character : .

Step 3: Tweet sentiment analysis

We apply sentiment analysis using textblob to identify the polarity and subjectivity scores of each tweet within the range [0,1]. Since pyspark does not have a built-in function for that, we used the user-defined function udf module to apply the textblob function.

Step 4: Run the main function

We first create an empty SparkSession , we connect it to the TCP socket we opened in Part 1, and we load the batches with the tweet data locally. As soon as we receive the batch from the socket, we preprocess the received data, and then we apply the text classification to each tweet to define its polarity and subjectivity. Then, we collect all the tweets and save them in one file every minute (60 seconds) for more efficient reads. The format of the saved file is parquet and for loading it we download the ParquetFileViewer.exe

Final thoughts

Congratulations! You made it here!

In this tutorial, we used Spark Structured Streaming to process and apply sentiment analysis of tweets that contain a specific keyword. You can change the keyword and query tweets related to topics of your interest. Just experiment with different keywords and get some insights about them!

Until next time!