Complete Introduction to PySpark-Part 2
By: Himanshu Sharma
Tags: Exploratory Data Analysis, Data Analysis, Pyspark, Python, Data Science

Complete Introduction to PySpark-Part 2

Exploratory Data Analysis using PySpark

Photo by Markus Spiske on Unsplash

Exploratory Data Analysis

Exploratory Data Analysis is the most crucial part, to begin with whenever we are working with a dataset. It allows us to analyze the data and let us explore the initial findings from data like how many rows and columns are there, what are the different columns, etc. EDA is an approach where we summarize the main characteristics of the data using different methods and mainly visualization.

Let’s start EDA using PySpark, before this if you have not yet installed PySpark, kindly visit the link below and get it configured on your Local Machine.

Importing Required Libraries and Dataset

Once we have configured PySpark on our machine we can use Jupyter Notebook to start exploring it. In this article, we will perform EDA operations using PySpark, for this we will using the Boston Dataset which can be downloaded Kaggle. Let’s start by importing the required libraries and loading the dataset.

#importing Required Libraries

import findspark

findspark.init() import pyspark # only run after findspark.init()

from pyspark.sql import SparkSession

from pyspark.sql import SQLContext #Creating a pyspark session

spark = SparkSession.builder.getOrCreate() #Importing Dataset

df = spark.read.csv('Boston.csv', inferSchema=True, header=True)

df.show(5)

Boston Dataset(Source: By Author)

Starting the EDA

There are different functions defined under pyspark which we can use for Exploratory Data Analysis, let us explore some of these functions and see how useful they are.

Schema

Schema is similar to the Info() function of pandas dataframe. It shows us the information about all the columns which are there in the dataset.

df.printSchema()

Schema(Source: By Author)

2. Describe

Describe function is used to display the statistical properties of all the columns in the dataset. It shows us values like Mean, Median, etc. for all the columns. In PySpark we need to call the show() function every time we need to display the information it works just like the head() function of python.

df.describe().show()

Statistical Properties(Source: By Author)

Similarly, we can use describe function column-wise also.

df.describe('AGE').show()

Describe Column Wise(Source: By Author)

3. Filter

The filter function is used to filter the data using different user-defined conditions. Let us see how we can use it accordingly.

#Filtering data with Indus=7.07

df.filter(df.INDUS==7.07).show()

Filter1(Source: By Author)

Similarly, we can use multiple filters in a single line of code.

df.filter((df.INDUS==7.07) & (df.MEDV=='High')).show()

Filter2(Source: By Author)

4. GroupBy and Sorting

PySpark inbuilt functions can be used to Group the data according to user requirements and also sorts the data as required.

df.groupBy('MEDV').count().show()

GroupBy(Source: By Author)

df.sort((df.TAX).desc()).show(5)

Sorting(Source: By Author)

5. Select & Distinct

The select function is used to select different columns while a distinct function can be used to select distinct values of that column.

df.select('MEDV').distinct().count()

Select and Distinct(Source: By Author)

6. WithColumn

WithColumn function is used to create a new column by providing certain conditions for the new columns and defining the name of the new column.

#Creating New column with values from Age column divided by 2

df.withColumn('HALF_AGE', df.AGE/2.0). select('AGE','HALF_AGE') .show(5)

WithColumn(Source: By Author)

In this article, we covered some major functions defined under PySpark which we can use for Exploratory Data Analysis and Understanding the data we are working on.

Go ahead, try these functions with different datasets, and if you face any problem let me know in the response section.

Before You Go

Thanks for reading! If you want to get in touch with me, feel free to reach me on hmix13@gmail.com or my LinkedIn Profile. You can view my Github profile for different data science projects and packages tutorial. Also, feel free to explore my profile and read different articles I have written related to Data Science.