PySpark on macOS: installation and use
By: René-Jean Corneille
Tags: Spark, Python, Data, Hadoop, Data Science

Spark is a very popular framework for data processing. It has slowly taken over the use of Hadoop for data analytics. In memory processing can yield up to 100x speed compared to Hadoop and MapReduce. One of the main advantages of Spark is that no more need to write map reduce jobs. Moreover, the spark engine is compatible with a large number of data sources (txt, json, xml, sql and nosql data stores). Spark is with Hadoop, SQL, Python and R one of the most sought after skills for data scientists.

A spark application is made of:

several execution processes which perform the data processing task.

a driver process: which is responsible for managing the resources allocated to the executors and distribute the data processing work load among the execution processes.

Users interact with the driver trough their code. Spark is initially written in Scala but also has APIs in other languages: R, Java and more importantly Python. Spark is meant to be run on a cluster of machines but can also be run locally as driver and executors are merely processes. This can be useful to prototype applications locally before sending them to the cloud. Google Cloud DataProc is (among other solutions) a very convenient tool to launch Spark jobs on the cloud.

Setting up Spark on MacOS does not have to be a pain. I have seen many posts on the topic, but I have not really been satisfied and often found myself not understand the reason behind some steps. So I decided to write my own post on the topic:

Install dependencies:

A very convenient way to install dependencies is with homebrew (in my opinion the best package manager for MacOS).

To install homebrew, you only need to copy this in your terminal:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Java

First we need to install Java as Spark is written in Scala, which is a Java Virtual Machine language. You can install the last version of Java with the command:

brew cask install java

You can always get information on the package using:

brew cask info java

You should get something that looks like this:

Scala

As Spark is written in Scala, we need to install it. If you get info on Spark using:

brew info apache-spark

You should get a result similar to this:

You notice that Java is a dependency. We install Scala with the command:

brew install scala

Spark

Finally, Spark can be installed on the system

brew install apache-spark

After installing the packages, it is good to check your system with

brew doctor

if you have: “Your system is ready to brew.” you can go to the next step. If you have any message you can follow the instructions given by the console.

2. Installing pySpark

I assume that you already have python 3 installed on your computer. If not installed already, you can do so with homebrew

brew install python3

Then you need to install the python Spark API pySpark:

pip3 install pyspark

3. Setting up environment

Almost done! Now you need to define several environment variables and declare paths so that the Spark driver is accessible through pySpark.

First of all we need to declare which java version to use. I have two versions on my system. You can do so by opening the .bashrc file with nano:

nano .bashrc

The java version can be declared with the following environment variable:

export JAVA_HOME=/Library/java/JavaVirtualMachines/adoptopenjdk-8.jdk/contents/Home/

and the java runtime environment is declared as follows:

export JRE_HOME=/Library/java/JavaVirtualMachines/openjdk-13.jdk/contents/Home/jre/

Now two variables must be declared for Spark:

export SPARK_HOME=/usr/local/Cellar/apache-spark/2.4.4/libexec

export PATH=/usr/local/Cellar/apache-spark/2.4.4/bin:$PATH

The first defines where your Spark libraries are installed and the second make visible your binaries to your shell (in particular the pypspark executable that we use later). Finally, PySpark must be configured:

export PYSPARK_PYTHON=/usr/local/bin/python3

export PYSPARK_DRIVER_PYTHON=jupyter

export PYSPARK_DRIVER_PYTHON_OPTS='notebook'

You can find your python3 distribution using the command:

which python3

which is used to define the variable PYSPARK_PYTHON. In order to be able to start a notebook with the command pyspark we define the two last variables: PYSPARK_DRIVER_PYTHON and PYSPARK_DRIVER_PYTHON_OPS.

You are all setup. The command

pyspark

will startup a new jupyter notebook session. The Spark UI can be consulted at the address:

http://localhost:4040

It looks like this: