How to build machine learning model at large scale with Apache Spark and LightGBM?
Tags: Apache Spark, Spark, Machine Learning, Predictive Analytics, Data Science

Image credit iStockPhoto.com

While it’s easy to train a machine learning model on Jupyter notebook on personal computer with relatively small static dataset, challenge appears when it is to be deployed in real environment inside financial institution where humongous amount of transactional data is residing in Hadoop or data lakes. In this post, I am going to show you how one can leverage Apache Spark to implement the core part of the building and training model inside Spark environment using LightGBM, which is considered to be a fast implementation of Gradient Boosting algorithm being widely used among data science community.

Apache Spark, is an in-memory clustered runtime environment to store and process data at speed thousands of time faster than accessing it from SSD or hard disks. Assuming the dataset that we want to work with is residing in Hadoop, S3, database or local file, the first task will be to have it accessible inside Spark cluster as Spark dataframe.

Fraud detection is an activity that every financial institution issuing credit cards is doing today with large scale dataset generated as result of millions of transactions happening on their platform everyday. In our case, I imported the dataset availed by Machine Learning Group of ULB in CSV file format as Spark dataframe on Spark cluster with 6gb RAM hosted by Databricks community edition. Here is the guide on adding data to Databricks.

# File location and type

file_location = "/FileStore/tables/creditcard.csv"

file_type = "csv"



# CSV options

infer_schema = "true"

first_row_is_header = "true"

delimiter = ","



df = spark.read.format(file_type) \

.option("inferSchema", infer_schema) \

.option("header", first_row_is_header) \

.option("sep", delimiter) \

.load(file_location)

The dataset contains around 300,000 rows consisting 31 variables related to European Credit Card holder’s transactions out of which 28 are numeric variables derived by performing Principal Component Analysis on some unrevealed original parameters. The remaining three variables are Amount of transaction, time of transaction in seconds relative to first transaction and label of transaction indicating whether its genuine or fraudulent. The dataset might look small when we are talking about large scale, but just by choosing to use Apache Spark for training the model even with small dataset, one can be rest assured that the same code will scale seamlessly to arbitrarily large dataset that might be hosted on HDFS, S3, NoSQL database or data lakes, as long as it is accessible in the form of Spark data frame to model.

The dataset is heavily imbalanced in terms of number of samples marked as fraudulent vs genuine. It has only small number of 492 samples pertaining to the transactions identified as fraudulent as compared to the overall number of rows in the dataset. You can access this dataset on Kaggle.

The next step is to select columns from this dataframe that we want to use as input variables and target variable while training our model. While building production grade ML model, it is often a good practise to perform data transformation steps like converting categorical columns into numeric columns using Label encoder or One Hot Encoder as stages of a pipeline, which can then be used for transforming training, validation or test data without leaking knowledge of test cases to training data (while performing transformations like standardized scaling) and high maintainability of code.

feature_cols = ["V" + str(i) for i in range(1,29)] + ["Amount"]

assembler = VectorAssembler(inputCols=feature_cols, outputCol="features")

stages = [assembler]

Next, we can add instance of LightGBMClassifier implementation to the pipeline. The LightGBMClassifier class is availalbe in MMLSpark library maintained as open source project by Microsoft Azure team. It can be included as library to the cluster by following steps suggested on this page. I added this library by searching package with name mmlspark from Maven option, selected version 0.17 of it and installed it in the cluster. After that, importing LightGBMClassifier from mmlspark in Python worked.

While using LightGBM, it’s highly important to tune it with optimal values of hyperparameters such as number of leaves, max depth, number of iterations etc. The performance of model trained on same data can vary greatly if trained with different values of hyperparameters.

Often times, this requires setting up a cross-validation experiment and using some hyper parameter value space exploration strategy such as full grid search, random search, Bayesian optimization or Tree-structured Parzen Estimators algorithm for finding out the optimal value of hyper parameters that maximize the performance of the model on validation dataset by minimizing some pre-defined error metric such as binary error or maximising some score such as Area under ROC or F1 measure.

This exploration can run long, depending on the size of parameter value space that we want to explore and the level of performance that we expect from our model. In this case, I decided to tune 7 hyperparameters of LightGBM model. Many of these parameters are real valued, meaning the parameter space to be explored is infinitely deep.

To gain access to uninterrupted computation time to explore this hyperparameter space by evaluating different combinations of hyperparameter values while using some optimization strategy to find out near optimal value of hyperparameters in relatively small number of evaluation rounds, I made use of Hyperopt library inside Kaggle’s execution environment in Python with this same dataset. I used Tree-structured Parzen Estimators algorithm to find out optimal values for hyperparameters after performing 200 evaluations of combinations of hyperparameters values. Here is the notebook where I determined the values of hyperparameters to be used while training this model inside Spark.

In community edition of Databricks, I got 2 hours of uninterrupted access to cluster. If you have access to longer uninterrupted time to Spark cluster, you may also use ParamGridBuilder and CrossValidator classes inside Spark to explore and evaluate hyperparameter values, as shown in the example code below. In case you want to be doing entire Hyperparameter tuning here in more efficient way that performing entire grid search, you can refer to these guides by Hyperopt and DataBricks, which shows how to use Spark cluster with Hyperopt.

Here I have used majority of the best hyperparameter values that I derived from the model tuning conducted on Kaggle’s python environment and tuned value of lambda L2 regularization parameter as an example inside Spark using CrossValidator for performing Grid Search.

Note that I have set the flag isUnbalance=True in LightGBMClassifier, so that it can take care of imbalance in the dataset that we discussed earlier.

best_params = {

'bagging_fraction': 0.8,

'bagging_freq': 1,

'eval_metric': 'binary_error',

'feature_fraction': 0.944714847210862,

'lambda_l1': 1.0,

'lambda_l2': 45.0,

'learning_rate': 0.1,

'loss_function': 'binary_error',

'max_bin': 60,

'max_depth': 58,

'metric': 'binary_error',

'num_iterations': 379,

'num_leaves': 850,

'objective': 'binary',

'random_state': 7,

'verbose': None

} lgb = LightGBMClassifier(

learningRate=0.1,

earlyStoppingRound=100,

featuresCol='features',

labelCol='label',

isUnbalance=True,

baggingFraction=best_params["bagging_fraction"],

baggingFreq=1,

featureFraction=best_params["feature_fraction"],

lambdaL1=best_params["lambda_l1"],

# lambdaL2=best_params["lambda_l2"],

maxBin=best_params["max_bin"],

maxDepth=best_params["max_depth"],

numIterations=best_params["num_iterations"],

numLeaves=best_params["num_leaves"],

objective="binary",

baggingSeed=7

) paramGrid = ParamGridBuilder().addGrid(

lgb.lambdaL2, list(np.arange(1.0, 101.0, 10.0))

).build() evaluator = BinaryClassificationEvaluator(labelCol="label",metricName="areaUnderROC") crossValidator = CrossValidator(estimator=lgb,

estimatorParamMaps=paramGrid,

evaluator=evaluator,

numFolds=2)

stages += [crossValidator]

pipelineModel = Pipeline(stages=stages)

The next step is to split the dataset into training and testing datasets. We will use the train dataset to fit the pipeline that we just created consisting of steps like feature assembling and model training. Then we will use this pipeline to transform the test dataset to generate predictions.

train, test = df.randomSplit([0.8, 0.2], seed=7)

model = pipelineModel.fit(train)

preds = model.transform(test)

Once we have the predictions from the test dataset, we can use them to measure the performance of our model. Spark provides BinaryClassificationEvaluator that can calculate Area under ROC metric.

To calculate other relevant metrics like precision, recall and F1 score, we can make use of the predicted labels and actual labels of our test dataset.

binaryEvaluator = BinaryClassificationEvaluator(labelCol="label") print ("Test Area Under ROC: " + str(binaryEvaluator.evaluate(preds, {binaryEvaluator.metricName: "areaUnderROC"}))) #True positives

tp = preds[(preds.label == 1) & (preds.prediction == 1)].count() #True negatives

tn = preds[(preds.label == 0) & (preds.prediction == 0)].count() #False positives

fp = preds[(preds.label == 0) & (preds.prediction == 1)].count() #Falsel negatives

fn = preds[(preds.label == 1) & (preds.prediction == 0)].count() print ("True Positives:", tp)

print ("True Negatives:", tn)

print ("False Positives:", fp)

print ("False Negatives:", fn)

print ("Total", preds.count()) r = float(tp)/(tp + fn) print ("recall", r) p = float(tp) / (tp + fp) print ("precision", p) f1 = 2 * p * r /(p + r) print ("f1", f1)

In this case, it received AUC-ROC score of 0.93 and F1 score of 0.70. In Kaggle notebook where I also used the SMOTE to balance the dataset before using it for training, it received the AUC-ROC score of 0.98 and F1 score near 0.80. I performed 200 evaluations for combinations of hyperparameter values in Kaggle environment. If we want to further increase the performance of our model, then obvious next thing to try would be to implement SMOTE in Spark for balancing training dataset before fitting the pipeline with it. We can also explore the hyperparameter space deeper by performing a larger number of evaluations of parameter value combinations.

Apart from training a high performant model like LightGBM on Spark, another big challenge data science teams commonly face is to manage the life cycle of preparing data, selecting the model, training, tuning, saving the best parameter values, deploying the trained model and accessing the output predictions as API. MLFlow is one open-source solution that tries to address this problem. I suggest the interested reader learn more about it. I might also write a post about how it can be integrated in the workflow for setting up an ML model building, training and deployment project just like the one that I demonstrated here using Spark on Databricks platform.

You can refer to the exact steps that I performed on Databricks in this notebook hosted on GitHub repo. Here is another notebook, where I conducted parameter tuning experiments for this dataset on Kaggle.

I hope you found this post insightful to get started with implementing machine learning on Apache Spark with LightGBMClassifier. If you have some questions while trying this out yourself, feel free to ask in comments.