How to Get Started with PySpark. PySpark is a Python API to using Spark... | by Sanket Gupta | Towards Data Science

PySpark is a Python API to using Spark, which is a parallel and distributed engine for running big data applications. Getting started with PySpark took me a few hours — when it shouldn’t have — as I had to read a lot of blogs/documentation to debug some of the setup issues. This blog is an attempt to help you get up and running on PySpark in no time!
UPDATE: I have written a new blog post on PySpark and how to get started with Spark with some of the managed services such as Databricks and EMR as well as some of the common architectures. It is titled Moving from Pandas to Spark. Check it out if you are interested to learn more!
These steps are for Mac OS X (I am running OS X 10.13 High Sierra), and for Python 3.6.
You can install Anaconda and if you already have it, start a new conda environment using conda create -n pyspark_env python=3 This will create a new conda environment with latest version of Python 3 for us to try our mini-PySpark project. Activate the environment with source activate pyspark_env
You could try using pip to install pyspark but I couldn’t get the pyspark cluster to get started properly. Reading several answers on Stack Overflow and the official documentation, I came across this:
The Python packaging for Spark is not intended to replace all of the other use cases. This Python packaged version of Spark is suitable for interacting with an existing cluster (be it Spark standalone, YARN, or Mesos) - but does not contain the tools required to setup your own standalone Spark cluster. You can download the full version of Spark from the Apache Spark downloads page.
Using the link above, I went ahead and downloaded the spark-2.3.0-bin-hadoop2.7.tgz and stored the unpacked version in my home directory.
Several instructions recommended using Java 8 or later, and I went ahead and installed Java 10. This actually resulted in several errors such as the following when I tried to run collect() or count() in my Spark cluster:
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.: java.lang.IllegalArgumentException
My initial guess was it had to do something with Py4J installation, which I tried re-installing a couple of times without any help. Not many people were talking about this error, and after reading several Stack Overflow posts, I came across this post which talked about how Spark 2.2.1 was having problems with Java 9 and beyond. The recommended solution was to install Java 8.
So, install Java 8 JDK and move to the next step.
To tell the bash how to find Spark package and Java SDK, add following lines to your .bash_profile (if you are using vim, you can do vim ~/.bash_profile to edit this file)
export JAVA_HOME=$(/usr/libexec/java_home)export SPARK_HOME=~/spark-2.3.0-bin-hadoop2.7export PATH=$SPARK_HOME/bin:$PATHexport PYSPARK_PYTHON=python3
These commands tell the bash how to use the recently installed Java and Spark packages. Run source ~/.bash_profile to source this file or open a new terminal to auto-source this file.
Run pyspark command and you will get to this:
You could use command line to run Spark commands, but it is not very convenient. You can install jupyter notebook using pip install jupyter notebook , and when you run jupyter notebook you can access the Spark cluster in the notebook. You can also just use vim or nano or any other code editor of your choice to write code into python files that you can run from command line.
Run a small and quick program to estimate the value of pi to see your Spark cluster in action!
import randomNUM_SAMPLES = 100000000def inside(p): x, y = random.random(), random.random() return x*x + y*y < 1count = sc.parallelize(range(0, NUM_SAMPLES)).filter(inside).count()pi = 4 * count / NUM_SAMPLESprint(“Pi is roughly”, pi)
There are lot of things in PySpark to explore such as Resilient Distributed Datasets or RDDs (update: now DataFrame API is the best way to use Spark, RDDs talk about “how” to do tasks vs Dataframes which talk about “what” — this makes Dataframes much faster and optimized) and MLlib.
UPDATE JUNE 2021: I have written a new blog post on PySpark and how to get started with Spark with some of the managed services such as Databricks and EMR as well as some of the common architectures. It is titled Moving from Pandas to Spark. Check it out if you are interested to learn more! Thank you for reading.