Build an end-to-end Machine Learning Model with MLlib in pySpark. | by Nasir Safdari | Towards Data Science

In-Memory computation and Parallel-Processing are some of the major reasons that Apache Spark has become very popular in the big data industry to deal with data products at large scale and perform faster analysis. built on top of Spark, MLlib is a scalable Machine Learning library that delivers both high-quality algorithms and blazing speed. having great APIs for Java, Python, and Scala, it makes a top choice for Data Analysts, Data Engineers, and Data Scientists. MLlib consists of common learning algorithms and utilities, including classification, regression, clustering, collaborative filtering (matrix factorization), dimensionality reduction and etc.
In this article, we are going to build an end-to-end machine learning model using MLlib in pySpark. we are going to use a real world dataset from Home Credit Default Risk competition on kaggle. the objective of this competition was to identify if loan applicants are capable of repaying their loans based on the data that was collected from each applicant. the target variable is either 0 (applicants who were able to pay back their loans)or 1 (applicants who were NOT able to pay back their loans). it is a binary classification problem with a highly imbalanced target label. the distribution ratio is close to 0.91 to 0.09, with 0.91 being the ratio of the applicants who were able to pay back their loans and 0.09 being the ratio of the applicants who were not able to pay back their loans.
Let’s start by looking at the structure of our dataset:
#we use the findspark library to locate spark on our local machineimport findsparkfindspark.init('home Diredtory of Spark')from pyspark.sql import SparkSession# initiate our session and read the main CSV file, then we print the #dataframe schemaspark = SparkSession.builder.appName('imbalanced_binary_classification').getOrCreate()new_df = spark.read.csv('application_train.csv', header=True, inferSchema=True)new_df.printSchema()root |-- SK_ID_CURR: integer (nullable = true) |-- TARGET: integer (nullable = true) |-- NAME_CONTRACT_TYPE: string (nullable = true) |-- CODE_GENDER: string (nullable = true) |-- FLAG_OWN_CAR: string (nullable = true) |-- FLAG_OWN_REALTY: string (nullable = true) |-- CNT_CHILDREN: integer (nullable = true) |-- AMT_INCOME_TOTAL: double (nullable = true) |-- AMT_CREDIT: double (nullable = true) |-- AMT_ANNUITY: double (nullable = true) |-- AMT_GOODS_PRICE: double (nullable = true) |-- NAME_TYPE_SUITE: string (nullable = true) |-- NAME_INCOME_TYPE: string (nullable = true) |-- NAME_EDUCATION_TYPE: string (nullable = true) |-- NAME_FAMILY_STATUS: string (nullable = true) |-- NAME_HOUSING_TYPE: string (nullable = true) |-- REGION_POPULATION_RELATIVE: double (nullable = true)...
printSchema() only shows us the column names and its data type. we are going to drop the SK_ID_CURR column, rename the “TARGET” column to “label” and see the distribution of our target variable:
# Sk_ID_Curr is the id column which we dont need it in the process #so we get rid of it. and we rename the name of our # target variable to "label"drop_col = ['SK_ID_CURR']new_df = new_df.select([column for column in new_df.columns if column not in drop_col])new_df = new_df.withColumnRenamed('TARGET', 'label')new_df.groupby('label').count().toPandas()
we can visualize the distribution of the labels with matplotlib:
# let's have a look at the distribution of our target variable:# to make it look better, we first convert our spark df to a Pandasimport matplotlib.pyplot as pltimport seaborn as sns%matplotlib inlinedf_pd = new_df.toPandas()print(len(df_pd))plt.figure(figsize=(12,10))sns.countplot(x='label', data=df_pd, order=df_pd['label'].value_counts().index)
and here is how the data looks like in a Pandas dataframe format:
# let's see how everything look in Pandasimport pandas as pdpd.DataFrame(new_df.take(10), columns= new_df.columns)
now that we have some ideas about the general structure of our dataset, let’s continue with some data wrangling. first we check how many Categorical and Numerical features do we have. next we build a function that outputs essential information about the missing values in our dataset:
# now let's see how many categorical and numerical features we have:cat_cols = [item[0] for item in new_df.dtypes if item[1].startswith('string')] print(str(len(cat_cols)) + '  categorical features')num_cols = [item[0] for item in new_df.dtypes if item[1].startswith('int') | item[1].startswith('double')][1:]print(str(len(num_cols)) + '  numerical features')16  categorical features104  numerical features
here is how we get the table for missing info:
# we use the below function to find more information about the #missing valuesdef info_missing_table(df_pd):    """Input pandas dataframe and Return columns with missing value and percentage"""    mis_val = df_pd.isnull().sum() #count total of null in each columns in dataframe#count percentage of null in each columns    mis_val_percent = 100 * df_pd.isnull().sum() / len(df_pd)     mis_val_table = pd.concat([mis_val, mis_val_percent], axis=1)  #join to left (as column) between mis_val and mis_val_percent    mis_val_table_ren_columns = mis_val_table.rename(    columns = {0 : 'Missing Values', 1 : '% of Total Values'}) #rename columns in table    mis_val_table_ren_columns = mis_val_table_ren_columns[    mis_val_table_ren_columns.iloc[:,1] != 0].sort_values('% of Total Values', ascending=False).round(1)             print ("Your selected dataframe has " + str(df_pd.shape[1]) + " columns.\n"    #.shape[1] : just view total columns in dataframe      "There are " + str(mis_val_table_ren_columns.shape[0]) +                  " columns that have missing values.") #.shape[0] : just view total rows in dataframe    return mis_val_table_ren_columnsmissings = info_missing_table(df_pd)missings
there are 67 columns out of 121 that has missing values in them. and it doesn't show all of them in the image, but overall, most of these 67 columns have more than 50 percent missing values. so we are dealing with a lot of missing values. we are going to fill the numerical missing values with the average of each column and the categorical missing values with the most frequent category of each column. but first, let’s count the missing values in each column:
miss_counts = count_missings(new_df)miss_counts[('AMT_ANNUITY', 12), ('AMT_GOODS_PRICE', 278), ('NAME_TYPE_SUITE', 1292), ('OWN_CAR_AGE', 202929), ('OCCUPATION_TYPE', 96391), ('CNT_FAM_MEMBERS', 2), ('EXT_SOURCE_1', 173378), ('EXT_SOURCE_2', 660), ('EXT_SOURCE_3', 60965), ('APARTMENTS_AVG', 156061), ('BASEMENTAREA_AVG', 179943), ('YEARS_BEGINEXPLUATATION_AVG', 150007), ('YEARS_BUILD_AVG', 204488),...
we separate categorical and numerical columns with missing values:
# here we seperate missing columns in our new_df based on #categorical and numerical typeslist_cols_miss=[x[0] for x in miss_counts]df_miss= new_df.select(*list_cols_miss)#categorical columnscatcolums_miss=[item[0] for item in df_miss.dtypes if item[1].startswith('string')]  #will select name of column with string data typeprint("cateogrical columns_miss:", catcolums_miss)### numerical columnsnumcolumns_miss = [item[0] for item in df_miss.dtypes if item[1].startswith('int') | item[1].startswith('double')] #will select name of column with integer or double data typeprint("numerical columns_miss:", numcolumns_miss)
next we fill the missing values:
# now that we have seperated the columns based on categorical and #numerical types, we will fill the missing categiracl # values with the most frequent categoryfrom pyspark.sql.functions import rank,sum,coldf_Nomiss=new_df.na.drop()for x in catcolums_miss:                  mode=df_Nomiss.groupBy(x).count().sort(col("count").desc()).collect()[0][0]     print(x, mode) #print name of columns and it's most categories     new_df = new_df.na.fill({x:mode})# and we fill the missing numerical values with the average of each #columnfrom pyspark.sql.functions import mean, roundfor i in numcolumns_miss:    meanvalue = new_df.select(round(mean(i))).collect()[0][0]     print(i, meanvalue)     new_df=new_df.na.fill({i:meanvalue})
now that we don’t have missing values anymore in our dataset, let’s work on how to deal with the imbalanced classes. there are different methods to mitigate this problem. one way is to under sample the majority class or over sample the minority class to make a more balanced results. another way is to assign weights for each class to penalize the majority class by assigning less weight and boost the minority class by assigning bigger weight. we are going to create a new column in the dataset named “weights” and assign the inverse ratio of each class as weights. here is how it’s done:
# adding the new column weights and fill it with ratiosfrom pyspark.sql.functions import whenratio = 0.91def weight_balance(labels):    return when(labels == 1, ratio).otherwise(1*(1-ratio))new_df = new_df.withColumn('weights', weight_balance(col('label')))
and here is how it looks after adding the weight column:
the next step is Feature Engineering. pySpark has made it so easy that we do not need to do much for extracting features. here are the steps:
we apply StringIndexer() to assign indices to each category in our categorical columns.we apply OneHotEncoderEstimator() to convert categorical columns to onehot encoded vectors.and we apply VectorAssembler() to create a feature vector from all categorical and numerical features and we call the final vector as “features”.
we apply StringIndexer() to assign indices to each category in our categorical columns.
we apply OneHotEncoderEstimator() to convert categorical columns to onehot encoded vectors.
and we apply VectorAssembler() to create a feature vector from all categorical and numerical features and we call the final vector as “features”.
# we use the OneHotEncoderEstimator from MLlib in spark to convert #aech v=categorical feature into one-hot vectors# next, we use VectorAssembler to combine the resulted one-hot ector #and the rest of numerical features into a # single vector column. we append every step of the process in a #stages arrayfrom pyspark.ml.feature import OneHotEncoderEstimator, StringIndexer, VectorAssemblerstages = []for categoricalCol in cat_cols:    stringIndexer = StringIndexer(inputCol = categoricalCol, outputCol = categoricalCol + 'Index')    encoder = OneHotEncoderEstimator(inputCols=[stringIndexer.getOutputCol()], outputCols=[categoricalCol + "classVec"])stages += [stringIndexer, encoder]assemblerInputs = [c + "classVec" for c in cat_cols] + num_colsassembler = VectorAssembler(inputCols=assemblerInputs, outputCol="features")stages += [assembler]
now let’s put everything into a pipeline. here we are performing a sequence of transformations so we do all of them at once using a pipeline:
# we use a pipeline to apply all the stages of transformationfrom pyspark.ml import Pipelinecols = new_df.columnspipeline = Pipeline(stages = stages)pipelineModel = pipeline.fit(new_df)new_df = pipelineModel.transform(new_df)selectedCols = ['features']+colsnew_df = new_df.select(selectedCols)pd.DataFrame(new_df.take(5), columns=new_df.columns)
here is how our new dataset looks like after feature engineering:
for training, we first split the dataset into training and testing sets. then we start training using Logistic Regression because it performs well for binary classification problems.
# split the data into trainign and testin setstrain, test = new_df.randomSplit([0.80, 0.20], seed = 42)print(train.count())print(test.count())# first we check how LogisticRegression perform from pyspark.ml.classification import LogisticRegressionLR = LogisticRegression(featuresCol = 'features', labelCol = 'label', maxIter=15)LR_model = LR.fit(train)
we are going to plot the ROC curve for the training data to see how Logistic Regression performed and then we will use Area Under ROC curve, which is a standard metric for evaluating binary classification, as the metric to evaluate the models:
#plotting the ROC CurvetrainingSummary = LR_model.summaryroc = trainingSummary.roc.toPandas()plt.plot(roc['FPR'],roc['TPR'])plt.ylabel('False Positive Rate')plt.xlabel('True Positive Rate')plt.title('ROC Curve')plt.show()print('Training set ROC: ' + str(trainingSummary.areaUnderROC))
checking model’s performance on the testing set:
from pyspark.ml.evaluation import BinaryClassificationEvaluatorpredictions_LR = LR_model.transform(test)evaluator = BinaryClassificationEvaluator()print("Test_SET (Area Under ROC): " + str(evaluator.evaluate(predictions_LR, {evaluator.metricName: "areaUnderROC"})))Test_SET (Area Under ROC): 0.7111434396856681
0.711 is not a very bad result for Logistic Regression. next we try another model, Gradient Boosting Trees (GBT). it’s a very popular classification and regression method that uses ensembles of decision trees.
# next we checkout gradient boosting treesfrom pyspark.ml.classification import GBTClassifiergbt = GBTClassifier(maxIter=15)GBT_Model = gbt.fit(train)gbt_predictions = GBT_Model.transform(test)evaluator = BinaryClassificationEvaluator()print("Test_SET (Area Under ROC): " + str(evaluator.evaluate(gbt_predictions, {evaluator.metricName: "areaUnderROC"})))Test_SET (Area Under ROC): 0.7322019340889893
we were able to achieve a much better result, 0.732, using GBT. as a final strategy here, we will implement hyper-parameter tuning using grid search and after that we run cross validation to better improve the performance of GBT.
from pyspark.ml.tuning import ParamGridBuilder, CrossValidatorparamGrid = (ParamGridBuilder()             .addGrid(gbt.maxDepth, [2, 4, 6])             .addGrid(gbt.maxBins, [20, 30])             .addGrid(gbt.maxIter, [10, 15])             .build())cv = CrossValidator(estimator=gbt, estimatorParamMaps=paramGrid, evaluator=evaluator, numFolds=5)# Run cross validations.cvModel = cv.fit(train)gbt_cv_predictions = cvModel.transform(test)evaluator.evaluate(gbt_cv_predictions)CV_GBT (Area Under ROC) = 0.7368288195372332
the result was improved a little bit which means that we still can play around with hyper-parameter tuning to see if we can further improve the result.
in this project, we built an End-to-End Machine Learning model (binary classification with imbalanced classes). and we showed the power of Apache Spark’s MLlib and how it can be applied for end-to-end ML projects.
like always, the code and jupyter notebook is available on my Github.
Questions and Comments are highly appreciated.