Complete Introduction to PySpark-Part 4 | by Himanshu Sharma | Towards Data Science

Data Visualization plays an important role in data analysis because as soon as the human eyes see some charts or graphs they try finding the patterns in that graph.
Data Visualization is visually representing the data using different plots/graphs/charts to find out the pattern, outliers, and relation between different attributes of a dataset. It is a graphical representation of data.
We can perform data visualization using PySpark but before that, we need to set it up on our local machine. In order to install PySpark on your local machine and get a basic understanding of how PySpark works, you can go through the given below articles.
medium.com
towardsdatascience.com
After installing PySpark let us start by opening a jupyter notebook and loading the required libraries.
We will start by loading all the required libraries and creating the PySpark session.
import findsparkfindspark.init()import pyspark # only run after findspark.init()from pyspark.sql import SparkSessionfrom pyspark.sql import SQLContextimport seaborn as snsimport matplotlib.pyplot as pltspark = SparkSession.builder.getOrCreate()
In this article, we will perform Data Visualization using PySpark, for this we will using the Boston Dataset which can be downloaded Kaggle. Let’s load the data and start visualization.
df = spark.read.csv('Boston.csv', inferSchema=True, header=True)df.show(5)
Before creating visualizations we need to create a table of the dataset using SQL, in order to learn more about SQL operations using PySpark go through the link given below.
Creating a Table using SQL and running required queries to generate a pie chart accordingly.
#Creating Tabledf.registerTempTable('BostonTable')sqlContext = SQLContext(spark)#Running Querydf1 = sqlContext.sql("SELECT * from BostonTable").toPandas()df2 = sqlContext.sql("SELECT AGE, TAX from BostonTable where LSTAT < 2").toPandas()#Creating Visualizationfig = plt.pie(df2['AGE'], autopct='%1.1f%%', startangle=140,labels=df2['AGE'])plt.title('No of age group where lstat < 2')plt.show()
Similarly, we can run different queries for different outputs accordingly we can change the Pie Chart visualization.
A correlation matrix is used to analyze and visualize the relationship between different columns of a dataset. Let us see how we can create a correlation matrix using PySpark.
df1.corr().style.background_gradient(cmap='coolwarm').set_precision(2)
Similarly, we can create different types of visualizations like Box-Plot, Violin Plot, Bar-plot, etc. Some of the examples of these plots are given below.
#box plotplt.boxplot(df2)plt.show()
#Scatter Plotdf3 = sqlContext.sql("SELECT PTRATIO, TAX from BostonTable").toPandas()fig=plt.figure()ax=fig.add_axes([0,0,1,1])ax.scatter(df3['TAX'],df3['PTRATIO'], color='r')ax.set_xlabel('TAX')ax.set_ylabel('PTRATIO')ax.set_title('scatter plot')plt.title('PTRATIO Vs TAX')plt.show()
#Violoin Plotdf5 = sqlContext.sql("SELECT RM from BostonTable").toPandas()fig = plt.figure()ax = fig.add_axes([0,0,1,1])bp = ax.violinplot(df5['RM'])plt.title('Average number of rooms per dwelling')plt.show()
#3d Scatter Plotdf6 = sqlContext.sql("SELECT CHAS, NOX, RM from BostonTable").toPandas()fig = plt.figure()ax = plt.axes(projection='3d')ax.scatter(df6['CHAS'], df6['NOX'], df6['RM'])ax.set_title('CHAS Vs NOX Vs RM')plt.show()
This article provides you with the basic information about Data Visualization using the PySpark. Go ahead try these and if you face any difficulty please let me know in the response section.
Thanks for reading! If you want to get in touch with me, feel free to reach me on hmix13@gmail.com or my LinkedIn Profile. You can view my Github profile for different data science projects and packages tutorial. Also, feel free to explore my profile and read different articles I have written related to Data Science.