Natural Language Processing with Spark | by Suraj Malpani | Towards Data Science

This is an introductory tutorial on developing predictive machine learning models using PySpark. I am going to demonstrate the basics of Natural Language Processing (NLP) while utilizing the power of Spark. We will use PySpark; which is a Python API for Spark. The dataset for this tutorial is fetched from the ‘NLP with Disaster Tweets’ Kaggle competition. The full code is available on GitHub.
The data consists of tweets and our task is to predict which tweets are related to a disaster. This may improve the response time for several interested parties, such as Police Force, Fire Brigade or News Agencies, etc. We will be performing text classification, by building predictive machine learning models, which is a category of NLP. The following algorithms can help you get instigated in your Text Analytics or NLP endeavor and has numerous applications.
Text classification is the process of assigning tags or categories to text according to its content. It’s one of the fundamental tasks in Natural Language Processing (NLP) with broad applications such as sentiment analysis, topic labeling, spam detection, and intent detection. — From Monkeylearn
Let’s dive into the data without further delay. For that, we need to know a bit about Spark Dataframes. Now, we need to start a Spark session within Python to use Spark. We start the session using the following command, appName parameter, i.e. ‘nlp’ in this case could be of the user’s choice.
spark = SparkSession.builder.appName('nlp').getOrCreate()
Once we start the Spark Session, we will load the data using ‘spark.read.csv’ function. After loading the data files into the workspace, we need to perform pre-processing on the text data. The head of the dataframe looks as displayed in the below image. We will be working on the ‘text’ field to predict the ‘target’ field.
The following workflow details the process that we will follow for extracting features from the data. The descriptions of each stage follow the image.
1) Drop null values: We drop all the records having a null/na value in the text. We can use dropna() function for this purpose.
2) Remove numbers from the tweets: We use Regular expressions (regex) operators for further cleaning the text data. The following code drops all the numbers from the text. We are dealing with the words rather than numbers to recognize the disaster tweets.
regexp_replace(col(‘text’), ‘\d+’, ‘’)
3) Segregate the words: Then we break the tweets into individual words to analyze them. We use RegexTokenizer() for this purpose.
4) Remove stop words: We remove stop words from the segregated words using StopWordsRemover() function from the pyspark.ml library. Some of the examples of stop words are I, the, a, has, etc. As you can notice from these examples, these words don’t carry much information and thus we remove them from the analysis.
5) Create the features column: After removing the unimportant words from the data, we use CountVectorizer() function. This function converts words to a numeric vector to be able to feed it to a machine learning model.
Now our data is ready to be fed to predictive models. The data looks like this after finishing the above process.
Let me shed light on all the fields in the above dataframe; we started with the ‘text’ field. The ‘words’ display segregated words after performing the first 3 steps from our above workflow. The ‘filtered’ column shows words after removing stop words as described in step 4. The ‘features’ field is the numeric vector field after finishing step 5, this is the field we will be using for training machine learning models. The ‘target’ field is our predictor variable which shows whether the tweet is related to a disaster or not.
We separate the data obtained from the previous process into train and validate. We use the train dataframe for training the machine learning models and validate dataframe to validate their accuracy on the unseen data. There are several criteria for validating the classification models, we will use ROC and accuracy for our analysis. After validating, we will make predictions on the test data.
I will illustrate some of the common classification machine learning algorithms. I am assuming you are familiar with these algorithms and I will try not to bore you with these algorithms.
Naive Bayes model is one of the most common algorithms for text classification. Naive Bayes algorithm assumes that all the predictor variables are independent of each other. Simply put, it assumes that the presence of one particular feature is unrelated to any other feature in the data. This assumption is not always correct in real life, however, it makes sense in text classification. The following code is used for training and validating the Naive Bayes Model using PySpark.
from pyspark.ml.classification import NaiveBayesfrom pyspark.ml.evaluation import MulticlassClassificationEvaluator## Fitting the modelnb = NaiveBayes(modelType="multinomial",labelCol="label", featuresCol="features")nbModel = nb.fit(train)nb_predictions = nbModel.transform(validate)## Evaluating the modelevaluator = MulticlassClassificationEvaluator(labelCol="label", predictionCol="prediction", metricName="accuracy")nb_accuracy = evaluator.evaluate(nb_predictions)print("Accuracy of NaiveBayes is = %g"% (nb_accuracy))#Accuracy of NaiveBayes is = 0.803448
It is a variant of regression that uses a logistic function for modeling a binary outcome variable. The following code demonstrates how to train and validate Logistic Regression.
from pyspark.ml.classification import LogisticRegression## Fitting the modellr = LogisticRegression(featuresCol = 'features', labelCol = 'target', maxIter=10)lrModel = lr.fit(train)lrPreds = lrModel.transform(validate)## Evaluating the modelevaluator = MulticlassClassificationEvaluator(labelCol="label", predictionCol="prediction", metricName="accuracy")lr_accuracy = evaluator.evaluate(lrPreds)print("Accuracy of Logistic Regression is = %g"% (lr_accuracy))#Accuracy of Logistic Regression is = 0.768276
Decision trees essentially split the dataset based on columns trying to learn their behavior concerning the outcome variable, i.e. we segment the predictor space into simple regions. This article by Will Koehrsen could help understand decision trees and random forests in detail. We can use the following code for training and validating decision trees.
from pyspark.ml.classification import DecisionTreeClassifier## Fitting the modeldt = DecisionTreeClassifier(featuresCol = 'features', labelCol = 'target', maxDepth = 3)dtModel = dt.fit(train)dtPreds = dtModel.transform(validate)## Evaluating the modelevaluator = MulticlassClassificationEvaluator(labelCol="label", predictionCol="prediction", metricName="accuracy")dt_accuracy = evaluator.evaluate(dtPreds)print("Accuracy of Decision Trees is = %g"% (dt_accuracy))#Accuracy of Decision Trees is = 0.651034
Random forests are an ensemble machine learning method. They use bootstrapping techniques and are fundamentally a combination of numerous weak learners, specifically decision trees. Check the following code to learn to implement random forests in spark.
from pyspark.ml.classification import RandomForestClassifier## Fitting the modelrf = RandomForestClassifier(featuresCol = 'features', labelCol = 'target')rfModel = rf.fit(train)rfPreds = rfModel.transform(validate)## Evaluating the modelevaluator = MulticlassClassificationEvaluator(labelCol="label", predictionCol="prediction", metricName="accuracy")rf_accuracy = evaluator.evaluate(rfPreds)print("Accuracy of Random Forests is = %g"% (rf_accuracy))#Accuracy of Random Forests is = 0.581379
Gradient boosting tree is an ensemble method analogous to random forests. The approach to building trees differs from the random forest; each new tree built by gradient boosting tree attempts to correct errors made by the previous tree. We use GBTClassifier from the pyspark.ml.classification library for training and validating the data.
from pyspark.ml.classification import GBTClassifier## Fitting the modelgbt = GBTClassifier(maxIter=10)gbtModel = gbt.fit(train)gbtPreds = gbtModel.transform(validate)## Evaluating the modelevaluator = MulticlassClassificationEvaluator(labelCol="label", predictionCol="prediction", metricName="accuracy")gb_accuracy = evaluator.evaluate(gbtPreds)print("Accuracy of GBT is = %g"% (gb_accuracy))#Accuracy of GBT is = 0.681379
Now that we’ve trained and validated several models, we make predictions on unseen test data. We can retrain the model on all available training data and then make predictions. The following code demonstrates how to make predictions using the Gradient boosting model. However, you can repeat the same for any of the trained models and also the full code for prediction using other models is available on Github.
## Fitting the modelgbt = GBTClassifier(maxIter=10)gbtModel = gbt.fit(trainData)## Make predictionsgbtPreds = gbtModel.transform(testData)gbtPreds.select('id','prediction').show(5)+---+----------+| id|prediction|+---+----------+|  0|       0.0||  2|       0.0||  3|       1.0||  9|       0.0|| 11|       0.0|+---+----------+only showing top 5 rows
Once you’ve made the predictions, you can convert the dataframe to a CSV format and submit it to the competition. You are now armed to use Spark for developing Machine Learning models. There are myriad applications of Text Classification, such as identifying spam emails, tagging website/product content, etc. and the above algorithms are applicable for all such tasks. Hopefully, you find this useful and will let me know what you think. Happy Learning!
Thank you for reading. I hope you find this helpful. If you have any suggestions, please add them in the comments section. Feel free to connect with me on my Website or LinkedIn.
Spark documentation: https://spark.apache.org/docs/2.2.0/ml-classification-regression.htmlhttps://monkeylearn.com/text-classification/Kaggle Notebook: https://www.kaggle.com/palmer0/binary-classification-with-pyspark-and-mllibGithub Notebook: https://github.com/lp-dataninja/SparkML/blob/master/pyspark-nlp-kaggle.ipynb
Spark documentation: https://spark.apache.org/docs/2.2.0/ml-classification-regression.html
https://monkeylearn.com/text-classification/
Kaggle Notebook: https://www.kaggle.com/palmer0/binary-classification-with-pyspark-and-mllib
Github Notebook: https://github.com/lp-dataninja/SparkML/blob/master/pyspark-nlp-kaggle.ipynb