A Guide to exploit Random Forest Classifier in PySpark
By: Manusha Priyanjalee
Tags: Random Forest, Classification, Pyspark, Data Science, Machine Learning

A Guide to exploit Random Forest Classifier in PySpark

Photo by Adél Grőber on Unsplash

In this article, I am going to give you a step-by-step guide on how to use PySpark for the classification of Iris flowers with Random Forest Classifier.

I have used the popular Iris dataset and I have provided the link to the dataset at the end of the article. I used Google Colab for coding and I have also provided Colab notebook in Resources.

Pyspark is a Python API for Apache Spark and pip is a package manager for Python packages.

!pip install pyspark

With the above command, pyspark can be installed using pip.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('ml-iris').getOrCreate()

df = spark.read.csv('IRIS.csv', header = True, inferSchema = True)

df.printSchema()

First, I need to create an entry point into all functionality in Spark. SparkSession class is used for this. SparkSession.builder() creates a basic SparkSession. To set a name for the application use appName(name) . Here I have set ‘ml-iris’ as the application name. Setting a name is not necessary and if it is not set, a random name will be generated for the application. getOrCreate() creates a new SparkSession if there is no existing session. Otherwise, it gets the existing session.

spark.read.csv(“path”) is used to read the CSV file into Spark DataFrame. A Data Frame is a 2D data structure and it sets data in a tabular format. Iris dataset has a header, so I set header = True , otherwise, the API treats the header as a data record. inferSchema attribute is related to the column types. By default, inferSchema is false. It will give all columns as strings. Here I set inferSchema = True , so Spark goes through the file and infers the schema of each column. printSchema() will print the schema in a tree format.

Output:

import pandas as pd

pd.DataFrame(df.take(5), columns=df.columns).transpose()

pandas is a toolkit used for data analysis. Here df.take(5) returns the first 5 rows and df.columns returns the names of all columns. DataFrame.transpose() transpose index and columns of the DataFrame. It writes columns as rows and rows as columns.

Output:

numeric_features = [t[0] for t in df.dtypes if t[1] == 'double']

df.select(numeric_features).describe().toPandas().transpose()

df.dtypes returns names and types of all columns. Here we assign columns of type Double to numeric_features. select(numeric_features) returns a new Data Frame. describe() computes statistics such as count, min, max, mean for columns and toPandas() returns current Data Frame as a Pandas DataFrame.

Output:

Since we have a good idea about the dataset we are working with now, we can start feature transforming. Feature transforming means scaling, converting, and modifying features so they can be used to train the machine learning model to make more accurate predictions. For this purpose, I have used String indexer, and Vector assembler.

First, I have used Vector Assembler to combine the sepal length, sepal width, petal length, and petal width into a single vector column. Here the new single vector column is called features.

Output:

Then I have used String Indexer to encode the string column of species to a column of label indices. By default, the labels are assigned according to the frequencies. So, the most frequent species gets an index of 0.

Output:

As you can see, we now have new columns named labelIndex and features.