Complete Introduction to PySpark- Part 3
By: Himanshu Sharma
Tags: Data Science, Big Data, Python, Sql, Data Analysis

Complete Introduction to PySpark- Part 3

Performing SQL operations on Datasets using PySpark

Photo by Franki Chamaki on Unsplash

What is SQL (Structured Query Language)?

SQL is a language that is used to perform different operations on data like storing, manipulating, and retrieving. It works on relational databases in which data is stored in the form of rows and columns.

SQL commands can be classified into three types according to their properties:

DDL(Data Definition Language)

As the name suggests DDL commands are used to define the data. The commands which are included in DDL are CREATE, INSERT, TRUNCATE, DROP, etc.

2. DML(Data Manipulation Language)

Data Manipulation commands are used to alter and update the data according to user requirements. Some of the commands defined under DDL are ALTER, UPDATE, DELETE, etc.

3. DCL(Data Controlling Language)

In this, the commands defined are used for controlling the access of the database defined. Some of the commands defined under this are GRANT, REVOKE, etc.

Using PySpark for SQL Operations

In order to perform SQL operations using PySpark, we need to have PySpark installed on our local machine. If you have already installed it we can get started else go through the below links to install PySpark and perform some basic operations on DataFrame using PySpark.

Loading Required Libraries

After we have installed pyspark on our machine and configure it, we will open a jupyter notebook to start SQL operations. We will start by importing the required libraries and creating a PySpark session.

import findspark

findspark.init() import pyspark # only run after findspark.init()

from pyspark.sql import SparkSession

from pyspark.sql import SQLContext

spark = SparkSession.builder.getOrCreate()

Loading the Dataset

For performing SQL operations we will need a dataset. In this article, we will use Boston Dataset which can be easily downloaded using Kaggle, and will load it using PySpark.

df = spark.read.csv('Boston.csv', inferSchema=True, header=True)

df.show(5)

Dataset(Source: By Author)

Now let us start SQL operations on our dataset, we will start by creating a table and an Object of SQLContext which will be used to run queries on that table.

1. Creating Table

For creating a table we will need to use the register function of PySpark. Similarly, we will also create an object of SQLContext use to run queries on the table.

df.registerTempTable('BostonTable')

sqlContext = SQLContext(spark)

2. Select Query

Select Query is used for selecting the data according to user requirements. We can use select the whole table using “*” or we can pass the name of the columns separated by ”,” that we want to see.

#Select Whole Table(only three records because we used show(3))

sqlContext.sql('select * from BostonTable').show(3)

Select Table(Source: By Author)

#Select column using column names

sqlContext.sql('select _c0, CRIM, ZN, INDUS, CHAS from BostonTable').show(3)

Select Columns(Source: By Author)

3. Aggregate Functions

There are some predefined aggregate functions defined in SQL which can be used to select data according to user requirements. These functions are:

a. min()

b. max()

c. count()

d. sum()

e. var()

etc.

The syntax for the following functions is given below.

#Using max functions

sqlContext.sql('select max(AGE) from BostonTable').show()

max function(Source: By Author)

Similarly, we can use other functions to display output according to user requirements.

4. Conditional Queries

By using conditional queries we can generate outputs that follow a certain condition passed by the user. The most used condition expression is “where”.

sqlContext.sql('select CRIM, NOX, B from BostonTable where B = 396.9').show(3)

Conditional Data(Source: By Author)

We can use different supporting functions in conditional queries which helps in being more specific about the output and can help in running multiple conditions in a single query. These functions are:

a. having

b. and

c. or

d. then

e. between(used for range)

etc.

sqlContext.sql('select CRIM, NOX, B, RAD from BostonTable where RAD > 2 and B = 396.9').show(3)

Conditional Expression(Source: By Author)

Similarly, we can use different functions using the same syntax as given above.

5. Nested Query

We can have multiple queries running in the same line of code which is generally called a nested query. It is a complex form of query where we pass different conditions to generate the output according to user demand. Below given is an example of a nested query.

sqlContext.sql('select * from BostonTable where AGE between 40 and 50 and TAX not in (311,307)').show(3)

Nested Query(Source: By Author)

Similarly, you can try different nested queries according to the output you want.

This article provides you with the basic information about the SQL Queries using the PySpark. Go ahead try these and if you face any difficulty please let me know in the response section.

Before You Go

Thanks for reading! If you want to get in touch with me, feel free to reach me on hmix13@gmail.com or my LinkedIn Profile. You can view my Github profile for different data science projects and packages tutorial. Also, feel free to explore my profile and read different articles I have written related to Data Science.