How to Get Started with PySpark
By: Sanket Gupta
Tags: Python, API, Spark, Data, Programming

These steps are for Mac OS X (I am running OS X 10.13 High Sierra), and for Python 3.6.

1. Start a new Conda environment

You can install Anaconda and if you already have it, start a new conda environment using conda create -n pyspark_env python=3 This will create a new conda environment with latest version of Python 3 for us to try our mini-PySpark project.

Activate the environment with source activate pyspark_env

2. Install PySpark Package

You could try using pip to install pyspark but I couldn’t get the pyspark cluster to get started properly. Reading several answers on Stack Overflow and the official documentation, I came across this:

The Python packaging for Spark is not intended to replace all of the other use cases. This Python packaged version of Spark is suitable for interacting with an existing cluster (be it Spark standalone, YARN, or Mesos) - but does not contain the tools required to setup your own standalone Spark cluster. You can download the full version of Spark from the Apache Spark downloads page.

Using the link above, I went ahead and downloaded the spark-2.3.0-bin-hadoop2.7.tgz and stored the unpacked version in my home directory.

3. Install Java 8

Several instructions recommended using Java 8 or later, and I went ahead and installed Java 10. This actually resulted in several errors such as the following when I tried to run collect() or count() in my Spark cluster:

py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe. : java.lang.IllegalArgumentException

My initial guess was it had to do something with Py4J installation, which I tried re-installing a couple of times without any help. Not many people were talking about this error, and after reading several Stack Overflow posts, I came across this post which talked about how Spark 2.2.1 was having problems with Java 9 and beyond. The recommended solution was to install Java 8.

So, install Java 8 JDK and move to the next step.

4. Change ‘.bash_profile’ variable settings

To tell the bash how to find Spark package and Java SDK, add following lines to your .bash_profile (if you are using vim, you can do vim ~/.bash_profile to edit this file)

export JAVA_HOME=$(/usr/libexec/java_home)

export SPARK_HOME=~/spark-2.3.0-bin-hadoop2.7

export PATH=$SPARK_HOME/bin:$PATH

export PYSPARK_PYTHON=python3

These commands tell the bash how to use the recently installed Java and Spark packages. Run source ~/.bash_profile to source this file or open a new terminal to auto-source this file.

5. Start PySpark

Run pyspark command and you will get to this:

PySpark welcome message on running `pyspark`

You could use command line to run Spark commands, but it is not very convenient. You can install jupyter notebook using pip install jupyter notebook , and when you run jupyter notebook you can access the Spark cluster in the notebook. You can also just use vim or nano or any other code editor of your choice to write code into python files that you can run from command line.

6. Calculate Pi using PySpark!

Run a small and quick program to estimate the value of pi to see your Spark cluster in action!

import random

NUM_SAMPLES = 100000000

def inside(p):

x, y = random.random(), random.random()

return x*x + y*y < 1 count = sc.parallelize(range(0, NUM_SAMPLES)).filter(inside).count()

pi = 4 * count / NUM_SAMPLES

print(“Pi is roughly”, pi)

7. Next Steps

There are lot of things in PySpark to explore such as Resilient Distributed Datasets or RDDs and MLlib, I am new to these areas and still learning these areas, hope to write a follow-up blog on these soon!

Good luck~

P.S: For the book lovers out there! I am working on a book search/recommender engine using a lot of these concepts (http://www.thereadable.club/) I’d appreciate if you try it out and give any feedback. Thanks! :)