Artificial Neural Network Using PySpark | by Somesh Routray | Towards Data Science

Introduction:This is a continuation of the Pyspark blog series. Previously I’ve shared the implementation of a basic Linear Regression using PySpark.In this blog, I’ll be showing another interesting implementation of a neural network using PySpark for a binary class prediction use-case. This blog will not be having lots of preprocessing steps but will give you an idea to implement it distributed environment especially when you run code in clusters in databricks. For Databricks environment if you want to leverage the TensorFlow, horovod is too handy for this or you can refer to the distributed tensorflow as well. But Horovod is recommended if you use it on the top of GPU clusters while working in the industry. However, I will make this blog too precise to understand pyspark way of neural network implementation for the readers.
I have taken a banknote authentication dataset. There are 5 columns in the dataset such as f_1,f_2,f_3,f_4 as features, and Class as the label.
To implement a Neural network in PySpark , we can use MultilayerPerceptronClassifier. It is super easy to use, just Import it from pyspark.ml.classification. It is based on the feedforward artificial neural network. Nodes in the input layer represent the input data. The rest of the nodes map the inputs to the outputs by a linear combination of the inputs with the node’s weights w and bias b and applying activation functions. This can be represented as :
It is based on the multi-layer perceptron.
It is based on the multi-layer perceptron.
2. Sigmoid activation function is used in each Layer and Softmax activation function is used in the output layer.
3. It uses logistic loss function for optimization and Solver L-BFGS for optimization routine. but You can another solver like gd.
gd stands for Gradient Descent Algorithm.
l-bdgs stands for Limited-memory Broyden–Fletcher–Goldfarb–Shanno Algorithm
Drawback:
It lacks the customization features when you define ann using Tensorflow or PyTorch.
Read more about the scikit learn implementation as well.
Implementation
Load the dataset:
dataset = spark.read.csv(“data_banknote_authentication.csv”,header=True)
Generally when you import the data using pyspark , if the schema is not predefined, it will take string type. So let’s now type cast the columns to double type.
for col in dataset.columns:    dataset = dataset.withColumn(col,dataset[col].cast('double'))
A few null values are there. so let’s impute it using mean values.
imputed_col = ['f_{}'.format(i+1) for i in range(len(input_cols))]model = Imputer(strategy='mean',missingValue=None,inputCols=input_cols,outputCols=imputed_col).fit(dataset)impute_data = model.transform(dataset)
In PySpark, we represent data in terms of vectors before feeding it into the model. Idea is to represent the data in matrix format and Scale the data between 0,1.
assemble = VectorAssembler(inputCols=imputed_col, outputCol='assembled_features', handleInvalid='error')a_data = assemble.transform(impute_data)scaler = MinMaxScaler(min=0.0, max=1.0, inputCol='assembled_features', outputCol='features')s_data = scaler.fit(a_data).transform(a_data)
Let’s now split the data into train and test datasets, define the model and fit the training data into the model. Network layer I have used is 4,16,2
4 because there are 4 different input features and 2 is the output layers as it is a binary classification.
train_df,test_df = s_data.select('Class ','features').randomSplit([0.7,0.3],1213)mlpc=MultilayerPerceptronClassifier( featuresCol=’features’,labelCol=’Class ‘,layers = [4,16,2],\ maxIter=1000,blockSize=8,seed=7,solver=’gd’)ann = mlpc.fit(train_df)
Once it is done. You can check the accuracy. Use a method MulticlassClassificationEvaluator().
Now Transform the test data, and in the evaluator method define the accuracies like f1, accuracy, and many more.
pred = ann.transform(test_df)evaluator = MulticlassClassificationEvaluator(labelCol='Class ',predictionCol='prediction',metricName='f1')ann_f1 = evaluator.evaluate(pred)ann_f1
After running for 1000 iterations, f1 score is 0.8309986610958963.
Conclusion:
Try out it with different datasets like mnist dataset with hyperparameter tunings and check model performance. Please let me know your thoughts in the comment section. You can access the code and dataset here .
I will be back with another interesting topic and till then happy reading!
Please share your thoughts in the comment section. You can connect me with on Linkedln, Gmail.