Machine Learning with Apache Spark
By: Manik Malhotra
Tags: Machine Learning, Python, Spark, Sparkml, Big Data

Big data is part of our lives now and most companies collecting data have to deal with big data in order to gain meaningful insights from them. While we know complex neural networks work beautifully and accurately when we have a big data set, at times they are not the most ideal. In a situation where the complexity of prediction is high, however, the prediction does need to be fast and efficient. Therefore, we need a scalable machine learning solution.

Apache spark comes with SparkML. SparkML has great inbuilt machine learning algorithms which are optimised for parallel processing and hence are very time-efficient on Big data. In this article, we will take a simple example of SparkML pipeline for cleaning, processing and generating predictions on big data.

We will take the weather data of JFK airport and make try several inbuilt classifiers in SparkML. The data set contains columns like wind speed, humidity, station pressure, etc. and we will try to classify the wind direction based on other inputs.

Lets being by cleaning the data set using Spark. Please note, I will leave a link to my GitHub repo for this code so you don't have to copy it from here. However, I will explain the code in this article

from pyspark import SparkContext, SparkConf

from pyspark.sql import SparkSession

from pyspark.ml.feature import OneHotEncoder, StringIndexer, VectorAssembler, Normalizer, MinMaxScaler

from pyspark.ml.linalg import Vectors

from pyspark.ml import Pipeline

import random

from pyspark.sql.functions import translate, col # spark context

sc = SparkContext.getOrCreate(SparkConf().setMaster("local[*]")) spark = SparkSession \

.builder \

.getOrCreate() # create a dataframe out of it by using the first row as field names # and trying to infer a schema based on contents

df = spark.read.option("header", "true").option("inferSchema","true").csv('noaa-weather-data-jfk-airport/jfk_weather.csv') # register a corresponding query table. we do this to save the data #in memory and run our operations on it.

df.createOrReplaceTempView('df') # cleaning the data as it contains trailing charcters. Double is a #data type like float

# columns with no trailing charecters were straight converrted to double type, rest were first cleaned

df_cleaned = df \

.withColumn("HOURLYWindSpeed", df.HOURLYWindSpeed.cast('double')) \

.withColumn("HOURLYWindDirection", df.HOURLYWindDirection.cast('double')) \

.withColumn("HOURLYStationPressure", translate(col("HOURLYStationPressure"), "s,", "")) \

.withColumn("HOURLYPrecip", translate(col("HOURLYPrecip"), "s,", "")) \

.withColumn("HOURLYRelativeHumidity", translate(col("HOURLYRelativeHumidity"), "*", "")) \

.withColumn("HOURLYDRYBULBTEMPC", translate(col("HOURLYDRYBULBTEMPC"), "*", "")) \ # the cleaned columsn were now changed to double types

df_cleaned = df_cleaned \

.withColumn("HOURLYStationPressure", df_cleaned.HOURLYStationPressure.cast('double')) \

.withColumn("HOURLYPrecip", df_cleaned.HOURLYPrecip.cast('double')) \

.withColumn("HOURLYRelativeHumidity", df_cleaned.HOURLYRelativeHumidity.cast('double')) \

.withColumn("HOURLYDRYBULBTEMPC", df_cleaned.HOURLYDRYBULBTEMPC.cast('double')) \ # Filtering for clean data set with no nulls and wind speed not 0

df_filtered = df_cleaned.filter("""

HOURLYWindSpeed <> 0

and HOURLYWindSpeed IS NOT NULL

and HOURLYWindDirection IS NOT NULL

and HOURLYStationPressure IS NOT NULL

and HOURLYPressureTendency IS NOT NULL

and HOURLYPrecip IS NOT NULL

and HOURLYRelativeHumidity IS NOT NULL

and HOURLYDRYBULBTEMPC IS NOT NULL

""") # saving the cleaned data set into CSV

df_filtered.write.csv('clean_df.csv')

The code above is my pre-processing script which will give me a clean data frame to work with and try different machine learning approaches. After looking at the raw data I made the following observations which needed to be addressed in pre-processing:

The data type was in the string and needed to be changed to something readable by algorithms Certain columns had trailing characters like ‘s’ and ‘*’ We had a significant amount of null values and not usable values like 0 in the target column

The first part of the code deals with the trailing characters where needed and converts the other columns to ‘double’ type which is like a float value. The second part of the code converts the clean columns to double type. The third part of the code filters out all the null values from the predictors and 0 values from the target variables. Lastly, we save the file to a CSV as clean df which we will use for machine learning operations.

I have tried out a few classifiers in my main code to check which one works the best, however, in this article I will only an example of logistic regression. Again no need to copy the code from here as I will provide my GitHub repo link.

from pyspark import SparkContext, SparkConf

from pyspark.sql import SparkSession

from pyspark.ml.feature import OneHotEncoder, StringIndexer, VectorAssembler, Normalizer, MinMaxScaler

from pyspark.ml.classification import LogisticRegression

from pyspark.ml.evaluation import MulticlassClassificationEvaluator

from pyspark.ml.feature import Bucketizer

from pyspark.ml.stat import Correlation

from pyspark.ml.linalg import Vectors

from pyspark.ml import Pipeline

import random # read our clean csv

df_filtered = spark.read.csv('clean_df.csv') # vector assembler

vectorAssembler = VectorAssembler(inputCols=["HOURLYWindSpeed","","HOURLYStationPressure"],

outputCol="features")

df_pipeline = vectorAssembler.transform(df_filtered) # checking correlations

Correlation.corr(df_pipeline,"features").head()[0].toArray() # train test split

splits = df_filtered.randomSplit([0.8, 0.2])

df_train = splits[0]

df_test = splits[1] # discretize the value using the Bucketizer, where we split the #column in buckets from above 0, 180 and then infinity

bucketizer = Bucketizer(splits=[ 0, 180, float('Inf') ],inputCol="HOURLYWindDirection", outputCol="HOURLYWindDirectionBucketized") # after the bucketizer we do one hot enncoding

encoder = OneHotEncoder(inputCol="HOURLYWindDirectionBucketized", outputCol="HOURLYWindDirectionOHE") # funtion for ccuracy calculation

def classification_metrics(prediction):

mcEval = MulticlassClassificationEvaluator().setMetricName("accuracy") .setPredictionCol("prediction").setLabelCol("HOURLYWindDirectionBucketized")

accuracy = mcEval.evaluate(prediction)

print("Accuracy on test data = %g" % accuracy) # logistic regression

# defining the model

lr = LogisticRegression(labelCol="HOURLYWindDirectionBucketized", maxIter=10) # new vector assembler

vectorAssembler = VectorAssembler(inputCols=["HOURLYWindSpeed","HOURLYDRYBULBTEMPC"],

outputCol="features") # bew piplineline for lr

pipeline = Pipeline(stages=[bucketizer,vectorAssembler,normalizer,lr]) # predictions

model = pipeline.fit(df_train)

prediction = model.transform(df_test)

classification_metrics(prediction)

The code above uses some common functionalities of SparkMl which a data scientist working spark should know. These are:

Vector assembler: is basically use to concatenate all the features into a single vector which can be further passed to the estimator or ML algorithm Correlation: spark provides a handy tool to check correlations for better feature engineering Bucketizer: Bucketing is an approach for converting the continuous variables into a categorical variable. We provide a range of buckets in splits which lets us categorise and use classification algorithms OneHotEncoder: One hot encoding is a process by which categorical variables are converted into a form that could be provided to ML algorithms to do a better job in prediction. Pipeline: Pipeline functionality in Spark lets you define a certain set of processes and the order in which they should be executed. Pipelines can also be saved and used later which makes it a great tool for scalability and portability.

In my code repository, I have used several other models like the random forest, gradient boosted trees. I also have tried a regression problem while predicting the wind speed on the same dataset. Feel free to reach out in case you have queries.

Thanks!

Link to my GitHub repo:

https://github.com/manikmal/spark_ml