How to Speed Up Your Python Code through PySpark | by Angelica Lo Duca | Towards Data Science

When you deal with huge datasets, the bottleneck is not your code (I hope so...), but the time elapsed to perform some operations on your dataset. For this reason, it is very important to exploit some libraries that can speed up your code.
Apache Spark may help you. Apache Spark is an open source project, which can speed up workloads up to 100x with respect to standard technologies. It is meant to work into a distributed environment (cluster), but it can be also used locally, for example when there are multiple processor in your machine.
In practice, in Apache Spark there are two types of nodes: the master, which is the main computer of the cluster and many workers. The master organises the work and distributes among workers and then it retrieves results.
In this tutorial, I deal with the following aspects:
download and install Apache Spark
install PySpark to configure Python to work with Apache Spark
run a simple example
Apache Spark can be downloaded from its official Web site:
spark.apache.org
You can choose the Spark release and the package time. If you do not have any particular need, you can download the last version and the pre-build for Apache Hadoop. After downloading it, you can move it to your preferred directory and rename it with a shorter name.
Apache Spark can be used in combination with Python, through the PySpark package.
Apache Spark also needs Java installed. Compatible versions of Java range from 8 to 11. The other versions are not supported.
Now you can install PySpark, for example through the pip manager:
pip install pyspark
Once installed, you need to configure the SPARK_HOME and modify the PATH variables in your .bash_profile or .profile file. This file is hidden and is located in your home directory. You can open it and add at the end of the file the following lines of code:
export SPARK_HOME="/path/to/spark/spark"export PATH="$SPARK_HOME/python:$PATH"
You can save the file and launch the terminal. You can enter the following command:
pyspark
The pyspark terminal starts. To exit it, simply write quit() and press Enter.
As option, you can configure pyspark to work with Jupyter notebooks.
In this case, you can use the findspark package, which will search for you where is Spark. In practice the findspark package will read the SPARK_HOME directory from your profile file.
You can install findspark through the following command:
pip install findspark
Once, installed, you can launch Jupyter notebook and add at the beginning of your code the following lines:
import findsparkfindspark.init()
Now, you are ready to run your first pyspark example. Firstly, you can create a SparkContext, which corresponds to the master of your cluster. You can specify some configuration parameters, such as the App Name (myproject) and the url (local):
from pyspark import SparkContext, SparkConfconf = SparkConf().setMaster("local").setAppName("myproject")sc = SparkContext.getOrCreate(conf=conf)
The getOrCreate() function creates a new SparkContext, if it does not already exist, otherwise it retrieves the existing one. This is due to the fact that there can be only one running instance of SparkContext.
Now you can create a Spark DataFrame, which is designed to behave a lot like a SQL table. A Spark DataFrame can be created from the SparkContext object as follows:
from pyspark.sql import SparkSessionspark = SparkSession.builder.getOrCreate()
Now you can use the spark object to read a CSV file:
df = spark.read.csv("/path/to/your/csv/file",inferSchema=True, header=True)
The inferSchema=True parameter permits to recognise automatically the data types but requires more time. You can list the first rows of the DataFrame through the show() function:
df.show()
the result looks like a SQL table.
You can query the DataFrame through the classic SQL queries. To do it, you firstly must register the new DataFrame to the list of available tables:
df_sql = df.createOrReplaceTempView('df_name')
where df_name is the name you want use for your table in SQL. You can list all the available tables:
print(spark.catalog.listTables())
Now you can query your table through the following code:
query = 'SELECT count(*) FROM df_name'sc.sql(query)
In this tutorial, I have illustrated how to install and run Apache Spark and PySpark in your computer. In addition, I have illustrated a simple example on how to read a CSV file in to Spark DataFrame.
If you wanted to be updated on my research and other activities, you can follow me on Twitter, Youtube and and Github.