Sentiment Analysis with PySpark. One of the tools I’m deeply interested... | by Ricky Kim | Towards Data Science

One of the tools I’m deeply interested but haven’t had many chances to explore is Apache Spark. Most of the time, Pandas and Scikit-Learn is enough to handle the size of data I’m trying to build a model on. But that also means that I haven’t had a chance to deal with petabytes of data yet, and I want to be prepared for the case I’m faced with a real big-data.
I have tried some basic data manipulation with PySpark before, but only to a very basic level. I want to learn more and be more comfortable in using PySpark. This post is my endeavour to have a better understanding of PySpark.
Python is great for data science modelling, thanks to its numerous modules and packages that help achieve data science goals. But what if the data you are dealing with cannot be fit into a single machine? Maybe you can implement careful sampling to do your analysis on a single machine, but with distributed computing framework like PySpark, you can efficiently implement the task for large datasets.
Spark API is available in multiple programming languages (Scala, Java, Python and R). There are debates about how Spark performance varies depending on which language you run it on, but since the main language I have been using is Python, I will focus on PySpark without going into too much detail of what language should I choose for Apache Spark.
Spark has three different data structures available through its APIs: RDD, Dataframe (this is different from Pandas data frame), Dataset. For this post, I will work with Dataframe, and the corresponding machine learning library SparkML. I first decided on the data structure I would like to use based on the advice from the post in Analytics Vidhya. “Dataframe is much faster than RDD because it has metadata (some information about data) associated with it, which allows Spark to optimize query plan.” You can find a comprehensive introduction from the original post.
And there’s also an informative post on Databricks comparing different data structures of Apache Spark: “A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets”.
Then I figured out that I need to use SparkML instead SparkMLLib if I want to deal with Dataframe. SparkMLLib is used with RDD, while SparkML supports Dataframe.
One more thing to note is that I will work in local mode with my laptop. The local mode is often used for prototyping, development, debugging, and testing. However, as Spark’s local mode is fully compatible with the cluster mode, codes written locally can be run on a cluster with just a few additional steps.
In order to use PySpark in Jupyter Notebook, you should either configure PySpark driver or use a package called Findspark to make a Spark Context available in your Jupyter Notebook. You can easily install Findspark by “pip install findspark” on your command line. Let’s first load some of the basic dependencies we need.
*In addition to short code blocks I will attach, you can find the link for the whole Jupyter Notebook at the end of this post.
import findsparkfindspark.init()import pyspark as psimport warningsfrom pyspark.sql import SQLContext
First step in any Apache programming is to create a SparkContext. SparkContext is needed when we want to execute operations in a cluster. SparkContext tells Spark how and where to access a cluster. It is first step to connect with Apache Cluster.
try:    # create SparkContext on all CPUs available: in my case I have 4 CPUs on my laptop    sc = ps.SparkContext('local[4]')    sqlContext = SQLContext(sc)    print("Just created a SparkContext")except ValueError:    warnings.warn("SparkContext already exists in this scope")
The dataset I’ll use for this post is annotated Tweets from “Sentiment140”. It originated from a Stanford research project, and I used this dataset for my previous series of Twitter sentiment analysis. Since I already cleaned the tweets during the process of my previous project, I will use pre-cleaned tweets. If you want to know more in detail about the cleaning process I took, you can check my previous post: “Another Twitter sentiment analysis with Python-Part 2” .
df = sqlContext.read.format('com.databricks.spark.csv').options(header='true', inferschema='true').load('project-capstone/Twitter_sentiment_analysis/clean_tweet.csv')type(df)
df.show(5)
df = df.dropna()df.count()
After successfully loading the data as Spark Dataframe, we can take a peek at the data by calling .show(), which is equivalent to Pandas .head(). After dropping NA, we have a bit less than 1.6 million Tweets. I will split this into three parts; training, validation, test. Since I have around 1.6 million entries, 1% each for validation and test set will be enough to test the models.
(train_set, val_set, test_set) = df.randomSplit([0.98, 0.01, 0.01], seed = 2000)
Through my previous attempt at sentiment analysis with Pandas and Scikit-Learn, I learned that TF-IDF with Logistic Regression is quite a strong combination, and showed robust performance, as high as Word2Vec + Convolutional Neural Network model. So in this post, I will try to implement TF-IDF + Logistic Regression model with PySpark.
By the way, if you want to know more in detail about how TF-IDF is calculated, please check my previous post: “Another Twitter sentiment analysis with Python — Part 5 (Tfidf vectorizer, model comparison, lexical approach)”
from pyspark.ml.classification import LogisticRegressionlr = LogisticRegression(maxIter=100)lrModel = lr.fit(train_df)predictions = lrModel.transform(val_df)from pyspark.ml.evaluation import BinaryClassificationEvaluatorevaluator = BinaryClassificationEvaluator(rawPredictionCol="rawPrediction")evaluator.evaluate(predictions)
0.86! That looks good, maybe too good. Because I already tried the same combination of techniques with the same data in Pandas and SKLearn, I know that the result for unigram TF-IDF with Logistic Regression is around 80% accuracy. There can be some slight difference due to the detailed model parameters, but still, this looks too good.
And by looking at the Spark documentation I realised that what BinaryClassificationEvaluator evaluates is by default areaUnderROC.
And for binary classification, Spark doesn’t support accuracy as a metric. But I can still calculate accuracy by counting the number of predictions matching the label and dividing it by the total entries.
accuracy = predictions.filter(predictions.label == predictions.prediction).count() / float(val_set.count())accuracy
Now it looks more plausible, actually, the accuracy is slightly lower than what I have seen from SKLearn’s result.
There’s another way that you can get term frequency for IDF (Inverse Document Frequency) calculation. It is CountVectorizer in SparkML. Apart from the reversibility of the features (vocabularies), there is an important difference in how each of them filters top features. In case of HashingTF it is dimensionality reduction with possible collisions. CountVectorizer discards infrequent tokens.
Let’s see if performance changes if we use CountVectorizer instead of HashingTF.
It looks like using CountVectorizer has improved the performance a little bit.
In Scikit-Learn, n-gram implementation is fairly easy. You can define a range of n-grams when you call TfIdf Vectorizer. But with Spark, it is a bit more complicated. It does not automatically combine features from different n-grams, so I had to use VectorAssembler in the pipeline, to combine the features I get from each n-gram.
I first tried to extract around 16,000 features from unigram, bigram, trigram. This means I will get around 48,000 features in total. Then I implemented Chi-Squared feature selection to reduce the number of features to 16,000 in total.
And now I’m ready to run the function I defined above.
%%timetrigram_pipelineFit = build_trigrams().fit(train_set)predictions = trigram_pipelineFit.transform(val_set)accuracy = predictions.filter(predictions.label == predictions.prediction).count() / float(dev_set.count())roc_auc = evaluator.evaluate(predictions)# print accuracy, roc_aucprint "Accuracy Score: {0:.4f}".format(accuracy)print "ROC-AUC: {0:.4f}".format(roc_auc)
Accuracy has improved, but as you might have noticed, fitting the model took 4 hours! And this is mainly because of ChiSqSelector.
What if I extract 5,460 features each from unigram, bigram, trigram in the first place, to have around 16,000 features in total in the end, without Chi Squared feature selection?
%%timetrigramwocs_pipelineFit = build_ngrams_wocs().fit(train_set)predictions_wocs = trigramwocs_pipelineFit.transform(val_set)accuracy_wocs = predictions_wocs.filter(predictions_wocs.label == predictions_wocs.prediction).count() / float(val_set.count())roc_auc_wocs = evaluator.evaluate(predictions_wocs)# print accuracy, roc_aucprint "Accuracy Score: {0:.4f}".format(accuracy_wocs)print "ROC-AUC: {0:.4f}".format(roc_auc_wocs)
This has given me almost same result, marginally lower, but the difference is in the fourth digit. Considering it takes only 6 mins without ChiSqSelector, I definitely choose the model without ChiSqSelector.
And finally, let’s try this model on the final test set.
test_predictions = trigramwocs_pipelineFit.transform(test_set)test_accuracy = test_predictions.filter(test_predictions.label == test_predictions.prediction).count() / float(test_set.count())test_roc_auc = evaluator.evaluate(test_predictions)# print accuracy, roc_aucprint "Accuracy Score: {0:.4f}".format(test_accuracy)print "ROC-AUC: {0:.4f}".format(test_roc_auc)
Final test set accuracy is 81.22% with ROC-AUC 0.8862.
Through this post, I have implemented a simple sentiment analysis model with PySpark. Even though it might not be an advanced level use of PySpark, but I believe it is important to keep expose myself to new environment and new challenges. Exploring some basic functions of PySpark really sparked (no pun intended) my interest.
I am attending Spark London Meetup tomorrow (13/03/2018) for “Apache Spark: Deep Learning Pipelines, PySpark MLLib and models in Streams”. I can’t wait to explore deeper into PySpark world!!
Thank you for reading and you can find the Jupyter Notebook from the below link: