Please use Pyspark for Hyper-parameter tuning to binge-watch more
By: Rehan Ahmad
Tags: Fine Tuning, Pyspark, Machine Learning, Udf, Random Search

Please use Pyspark for Hyper-parameter tuning to binge-watch more

After months of chasing and scrutiny, finally, you got access to the data dump you have been waiting for. You are already done with the pipeline of necessary preprocessing steps, some feature engineering and some model selection. After a bit of experimenting, you picked your best algorithm to proceed with, execute the hyperparameter tuning part, fill up your big cup of coffee and you wait, and wait, and wait…….and wait for the results like a never-ending story. This is a vastly common phenomenon in any typical Data scientist/Machine learning engineer’s daily routine. I am sure it’s considered the most boring and monotonous task in the whole ML life cycle if there is any. I stumbled upon a cool way that could reduce this waiting time to a great extent. It’s pure magic. It’s like after reading this “You are a wizard Harry!”. Anyway, the magic’s got a name and it’s…..wait for it….. “Parallelization”. And the spell to use is Pyspark.

Hyperparameter Tuning is nothing but searching for the right set of hyperparameter to achieve high precision and accuracy. Optimising hyperparameters constitute one of the trickiest part of building the machine learning models. The primary aim of hyperparameter tuning is to find the sweet spot for the model’s parameters so that a better performance is obtained.

The 2 most common approaches to do hyperparameter tuning in a typical-classical (by this I mean the noobs who use Scikit-Learn for ML (No offence)) machine learning pipeline are

Grid search In Grid Search, we try every combination of a preset list of values of the hyper-parameters and evaluate the model for each combination. Random Search Random search is a technique where random combinations of the hyperparameters are used to find the best solution for the built model.

While less common in machine learning practice than grid search, random search has been shown to find equal or better values than grid search within fewer function evaluations for certain types of problems. It’s my pick as well for most of my projects. So, I explained the magic-using Random search in this post.

For the sake of budding Data scientists who won’t have a grid of clusters at their disposal, I used the Databricks Community Edition Free server with a toy example in the post. Here is a handy tutorial on how to set up a free cluster and play with it. Who would have thought that fine-tuning at scale can be achieved for free?

The idea of how

Spark does it’s magic the best with key-value pairs. The way it achieves parallelization is by sending all the data off with a particular key to a single node. A subset of keys and their associated data goes to each node in a cluster for processing. This is a problem for us Data Scientists. We need all the data on all the nodes to be able to do a random search and fit a model. Or any task for that matter. Focus on the following as it is a crucial trick. A street smart solution for this is to replicate all the data multiple times and assign a replication_id for each replication. This way we can group by replication_id and each replication with entire data can be sent to each node of a cluster. This neat little trick makes it possible for us to do our machine learning tasks at scale leveraging PySpark magic of parallelization. You might ask, how can we use standard machine learning code with pandas and sklearn in pyspark on a spark cluster. To enable data scientists to leverage the value of big data, Spark added a Python API in version 0.7, with support for user-defined functions. These user-defined functions operate one-row-at-a-time and thus suffer from high serialization and invocation overhead. Worry not, pandas_udf to the rescue. Pandas UDFs built on top of Apache Arrow bring you the best of both worlds — the ability to define low-overhead, high-performance UDFs entirely in Python. This functionality was introduced in the Spark version 2.3.1. More on this here.

Code

The following code can be run on a Databricks community Edition free spark cluster. Get it here.

Importing libraries necessary libraries.

from sklearn import datasets

import pandas as pd

from sklearn.ensemble import RandomForestClassifier

from sklearn.metrics import accuracy_score

import pyspark.sql.functions as F

import random

from pyspark.sql.types import *

from sklearn.model_selection import train_test_split

2. Making a toy classification example with 10000 records.

X,y = datasets.make_classification(n_samples=10000, n_features=4, n_informative=2, n_classes=2, random_state=1,shuffle=True)

3. Getting X and y into a single pandas data frame.

training_data = pd.DataFrame(X)

training_data['target'] = y

4. Converting pandas data frame to spark data frame.

train_spark_df = spark.createDataFrame(training_data)

train_spark_df = train_spark_df.toDF(*['c0', 'c1', 'c2', 'c3', 'target'])

5. Create a spark data frame with only one column of replication ids.

replication_df = spark.createDataFrame(pd.DataFrame(list(range(1,1001)),columns=['replication_id']))

6. Apply cross join between the 2 data frames to get our key-value pairs with full data as value and replication id as a key.

replicated_train_df = train_spark.crossJoin(replication_df)

7. Define an output schema for the UDF function to output(This is kinda expected by Pyspark)

outSchema = StructType([StructField('replication_id',IntegerType(),True),StructField('Accuracy',DoubleType(),True),StructField('num_trees',IntegerType(),True),StructField('depth',IntegerType(),True),StructField('criterion',StringType(),True)])

8. Now for the actual UDF function. It looks like a normal python function with an extra decorator on top of it.



def run_model(pdf): @F .pandas_udf(outSchema, F.PandasUDFType.GROUPED_MAP)def run_model(pdf): ## this is how we are picking random set of parameters.

num_trees = random.choice(list(range(50,500)))

depth = random.choice(list(range(2,10)))

criterion = random.choice(['gini','entropy']) replication_id = pdf.replication_id.values[0] # Split X and y

X = pdf[['c0', 'c1', 'c2', 'c3']]

y = pdf['target'] # Test train split

Xtrain,Xcv,ytrain,ycv = train_test_split(X, y, test_size=0.33, random_state=42) # Initilize model and fit

clf = RandomForestClassifier(n_estimators=num_trees, max_depth = depth, criterion =criterion)

clf.fit(Xtrain,ytrain) # Get the accuracies

accuracy = accuracy_score(clf.predict(Xcv),ycv) # Return result as a pandas data frame

res=pd.DataFrame({'replication_id':replication_id,'Accuracy':accuracy,'num_trees':num_trees,'depth':depth,'criterion':criterion}, index=[0])

return res

Please do some reading on how the “F.PandasUDFType.GROUPED_MAP” works here.

9. Once you have the UDF in place, now its time to apply it to our column.

results = replicated_train_df.groupby("replication_id").apply(run_model)

10. The above statement will not actually process the data but will save the instruction set to be applied on the column. The real processing starts when we request action on data like show(), count() etc.

results.sort(F.desc("Accuracy")).show()

Output:

The processing took 10.19 minutes. Just to compare how better is that with regular python, here is a pure Python equivalent of what we tried to achieve using sklearn’s inbuilt RandomizedSearchCV function for 1000 iterations.

# Load libraries

from scipy.stats import uniform

from sklearn import linear_model, datasets

from sklearn.model_selection import RandomizedSearchCV

import random

from sklearn.ensemble import RandomForestClassifier

import time X,y = datasets.make_classification(n_samples=10000, n_features=4, n_informative=2, n_classes=2, random_state=1,shuffle=True) n_estimators = [int(x) for x in range(50,501)]

max_depth = [int(x) for x in range(2,11)]

criterion = ['gini','entropy'] hyperparameters = dict(n_estimators=n_estimators, max_depth=max_depth, criterion= criterion) clf = RandomForestClassifier() clf = RandomizedSearchCV(clf, hyperparameters, random_state=1, n_iter=1000, verbose=0, n_jobs=-1) start_time = time.time()

best_model = clf.fit(X, y)

print("--- %s seconds ---" % (time.time() - start_time))

Output:

As you can see pure python took 38.3 minutes but pyspark took 10.2 minutes. This is quite an improvement already. Pyspark’s processing time will reduce even further and python takes even longer when you throw more data at it (Like making the total records 1000000 instead of 10000).

Here is a gist of all the code together for you copy-paste purpose.

May this give you the power vested in Pypark on your future embarkments of Machine Learning to not waste any more time than needed.

And about the binge-watching part. If you can save time on hyperparameter tuning, you can obviously spend more time on Netflix. Just saying ;)

About me

I am a Senior AI Expert at Wavelabs.ai. We at Wavelabs help you leverage Artificial Intelligence (AI) to revolutionize user experiences and reduce costs. We uniquely enhance your products using AI to reach your full market potential. We try to bring cutting edge research into your applications.

Feel free to explore more at Wavelabs.ai.