PySpark and SparkSQL Basics. How to implement Spark with Python... | by Pınar Ersoy | Towards Data Science

Python is revealed the Spark programming model to work with structured data by the Spark Python API which is called as PySpark.
This post’s objective is to demonstrate how to run Spark with PySpark and execute common functions.
Python programming language requires an installed IDE. The easiest way to use Python with Anaconda since it installs sufficient IDE’s and crucial packages along with itself.
With the help of this link, you can download Anaconda. After the suitable Anaconda version is downloaded, click on it to proceed with the installation procedure which is explained step by step in the Anaconda Documentation.
When the installation is completed, the Anaconda Navigator Homepage will be opened. In order to use Python, simply click on the “Launch” button of the “Notebook” module.
To be able to use Spark through Anaconda, the following package installation steps shall be followed.
Anaconda Prompt terminal
conda install pyspark
conda install pyarrow
After PySpark and PyArrow package installations are completed, simply close the terminal and go back to Jupyter Notebook and import the required packages at the top of your code.
import pandas as pdfrom pyspark.sql import SparkSessionfrom pyspark.context import SparkContextfrom pyspark.sql.functions import *from pyspark.sql.types import *from datetime import date, timedelta, datetimeimport time
First of all, a Spark session needs to be initialized. With the help of SparkSession, DataFrame can be created and registered as tables. Moreover, SQL tables are executed, tables can be cached, and parquet/JSON/CSV/Avro data formatted files can be read.
sc = SparkSession.builder.appName("PysparkExample")\    .config ("spark.sql.shuffle.partitions", "50")\    .config("spark.driver.maxResultSize","5g")\    .config ("spark.sql.execution.arrow.enabled", "true")\    .getOrCreate()
For detailed explanations for each parameter of SparkSession, kindly visit pyspark.sql.SparkSession.
A DataFrame can be accepted as a distributed and tabulated collection of titled columns which is similar to a table in a relational database. In this post, we will be using DataFrame operations on PySpark API while working with datasets.
You can download the Kaggle dataset from this link.
3.1. From Spark Data Sources
DataFrames can be created by reading text, CSV, JSON, and Parquet file formats. In our example, we will be using a .json formatted file. You can also find and read text, CSV, and Parquet file formats by using the related read functions as shown below.
#Creates a spark data frame called as raw_data.#JSONdataframe = sc.read.json('dataset/nyt2.json')#TXT FILES# dataframe_txt = sc.read.text('text_data.txt')#CSV FILES# dataframe_csv = sc.read.csv('csv_data.csv')#PARQUET FILES# dataframe_parquet = sc.read.load('parquet_data.parquet')
Duplicate values in a table can be eliminated by using dropDuplicates() function.
dataframe = sc.read.json('dataset/nyt2.json') dataframe.show(10)
After dropDuplicates() function is applied, we can observe that duplicates are removed from the dataset.
dataframe_dropdup = dataframe.dropDuplicates() dataframe_dropdup.show(10)
Querying operations can be used for various purposes such as subsetting columns with “select”, adding conditions with “when” and filtering column contents with “like”. Below, some of the most commonly used operations are exemplified. For the complete list of query operations, see the Apache Spark doc.
5.1. “Select” Operation
It is possible to obtain columns by attribute (“author”) or by indexing (dataframe[‘author’]).
#Show all entries in title columndataframe.select("author").show(10)#Show all entries in title, author, rank, price columnsdataframe.select("author", "title", "rank", "price").show(10)
5.2. “When” Operation
In the first example, the “title” column is selected and a condition is added with a “when” condition.
# Show title and assign 0 or 1 depending on titledataframe.select("title",when(dataframe.title != 'ODD HOURS', 1).otherwise(0)).show(10)
In the second example, the “isin” operation is applied instead of “when” which can be also used to define some conditions to rows.
# Show rows with specified authors if in the given optionsdataframe [dataframe.author.isin("John Sandford", "Emily Giffin")].show(5)
5.3. “Like” Operation
In the brackets of the “Like” function, the % character is used to filter out all titles having the “ THE ” word. If the condition we are looking for is the exact match, then no % character shall be used.
# Show author and title is TRUE if title has " THE " word in titlesdataframe.select("author", "title",dataframe.title.like("% THE %")).show(15)
5.4. “Startswith” — “ Endswith”
StartsWith scans from the beginning of word/content with specified criteria in the brackets. In parallel, EndsWith processes the word/content starting from the end. Both of the functions are case-sensitive.
dataframe.select("author", "title", dataframe.title.startswith("THE")).show(5)dataframe.select("author", "title", dataframe.title.endswith("NT")).show(5)
5.5. “Substring” Operation
Substring functions to extract the text between specified indexes. In the following examples, texts are extracted from the index numbers (1, 3), (3, 6), and (1, 6).
dataframe.select(dataframe.author.substr(1, 3).alias("title")).show(5)dataframe.select(dataframe.author.substr(3, 6).alias("title")).show(5)dataframe.select(dataframe.author.substr(1, 6).alias("title")).show(5)
Data manipulation functions are also available in the DataFrame API. Below, you can find examples to add/update/remove column operations.
6.1. Adding Columns
# Lit() is required while we are creating columns with exact values.dataframe = dataframe.withColumn('new_column', F.lit('This is a new column'))display(dataframe)
6.2. Updating Columns
For updated operations of DataFrame API, withColumnRenamed() function is used with two parameters.
# Update column 'amazon_product_url' with 'URL'dataframe = dataframe.withColumnRenamed('amazon_product_url', 'URL')dataframe.show(5)
6.3. Removing Columns
Removal of a column can be achieved in two ways: adding the list of column names in the drop() function or specifying columns by pointing in the drop function. Both examples are shown below.
dataframe_remove = dataframe.drop("publisher", "published_date").show(5)dataframe_remove2 = dataframe \ .drop(dataframe.publisher).drop(dataframe.published_date).show(5)
There exist several types of functions to inspect data. Below, you can find some of the commonly used ones. For a deeper look, visit the Apache Spark doc.
# Returns dataframe column names and data typesdataframe.dtypes# Displays the content of dataframedataframe.show()# Return first n rowsdataframe.head()# Returns first rowdataframe.first()# Return first n rowsdataframe.take(5)# Computes summary statisticsdataframe.describe().show()# Returns columns of dataframedataframe.columns# Counts the number of rows in dataframedataframe.count()# Counts the number of distinct rows in dataframedataframe.distinct().count()# Prints plans including physical and logicaldataframe.explain(4)
The grouping process is applied with GroupBy() function by adding column name in function.
# Group by author, count the books of the authors in the groupsdataframe.groupBy("author").count().show(10)
Filtering is applied by using the filter() function with a condition parameter added inside of it. This function is case-sensitive.
# Filtering entries of title# Only keeps records having value 'THE HOST'dataframe.filter(dataframe["title"] == 'THE HOST').show(5)
For every dataset, there is always a need for replacing, existing values, dropping unnecessary columns, and filling missing values in data preprocessing stages. pyspark.sql.DataFrameNaFunction library helps us to manipulate data in this respect. Some examples are added below.
# Replacing null valuesdataframe.na.fill()dataFrame.fillna()dataFrameNaFunctions.fill()# Returning new dataframe restricting rows with null valuesdataframe.na.drop()dataFrame.dropna()dataFrameNaFunctions.drop()# Return new dataframe replacing one value with anotherdataframe.na.replace(5, 15)dataFrame.replace()dataFrameNaFunctions.replace()
It is possible to increase or decrease the existing level of partitioning in RDD Increasing can be actualized by using the repartition(self, numPartitions) function which results in a new RDD that obtains the higher number of partitions. Decreasing can be processed with coalesce(self, numPartitions, shuffle=False) function that results in a new RDD with a reduced number of partitions to a specified number. For more info, please visit the Apache Spark docs.
# Dataframe with 10 partitionsdataframe.repartition(10).rdd.getNumPartitions()# Dataframe with 1 partitiondataframe.coalesce(1).rdd.getNumPartitions()
Raw SQL queries can also be used by enabling the “sql” operation on our SparkSession to run SQL queries programmatically and return the result sets as DataFrame structures. For more detailed information, kindly visit Apache Spark docs.
# Registering a tabledataframe.registerTempTable("df")sc.sql("select * from df").show(3)sc.sql("select \               CASE WHEN description LIKE '%love%' THEN 'Love_Theme' \               WHEN description LIKE '%hate%' THEN 'Hate_Theme' \               WHEN description LIKE '%happy%' THEN 'Happiness_Theme' \               WHEN description LIKE '%anger%' THEN 'Anger_Theme' \               WHEN description LIKE '%horror%' THEN 'Horror_Theme' \               WHEN description LIKE '%death%' THEN 'Criminal_Theme' \               WHEN description LIKE '%detective%' THEN 'Mystery_Theme' \               ELSE 'Other_Themes' \               END Themes \       from df").groupBy('Themes').count().show()
13.1. Data Structures
DataFrame API uses RDD as a base and it converts SQL queries into low-level RDD functions. By using the .rdd operation, a dataframe can be converted into RDD. It is also possible to convert Spark Dataframe into a string of RDD and Pandas formats.
# Converting dataframe into an RDDrdd_convert = dataframe.rdd# Converting dataframe into a RDD of string dataframe.toJSON().first()# Obtaining contents of df as Pandas dataFramedataframe.toPandas()
13.2. Write & Save to Files
Any data source type that is loaded to our code as data frames can easily be converted and saved into other types including .parquet and .json. For more save, load, write function details, please visit Apache Spark doc.
# Write & Save File in .parquet formatdataframe.select("author", "title", "rank", "description") \.write \.save("Rankings_Descriptions.parquet")
# Write & Save File in .json formatdataframe.select("author", "title") \.write \.save("Authors_Titles.json",format="json")
13.3. Stopping SparkSession
Spark Session can be stopped by running the stop() function as follows.
# End Spark Sessionsc.stop()
The code and Jupyter Notebook are available on my GitHub.
Questions and comments are highly appreciated!