Using pyspark with Jupyter on a local computer
By: Nimrod Milo
Tags: Spark, Python, Data Engineering

Installing Spark on Linux

This manual was tested on version 2.2.0 but should work on all versions. I’m assuming you already have Python and Java installed. In order to install Spark on your machine, follow the next steps:

Download the tar.gz file from Apache website (I’m assuming you are downloading to /opt):

wget https://www.apache.org/dyn/closer.lua/spark/spark-2.2.1/spark-2.2.1-bin-hadoop2.7.tgz extract the file and create a soft link to the folder:

tar -xvzf spark-2.2.1-bin-hadoop2.7.tgz

ln -s spark-2.2.1-bin-hadoop2.7 spark Verify the py4j version (we’ll need it to connect Spark and Jupyter):

ls -1 /opt/spark/python/lib/py4j* | awk -F "-" '{print $2}'

The output I got was 0.10.4, we will use it later as <<PY4J_VERSION>> Verify the Python path you use:

which python

The output I got was /home/nimrod/miniconda/envs/compass/bin/python2, We’ll use this as <<PYTHON_HOME>>

After completing the above, create a kernel json file:

mkdir -p ~/.local/share/jupyter/kernels/spark2-local

Edit the file:

vim ~/.local/share/jupyter/kernels/spark2-local/kernel.json

And add the following content (don’t forget to replace the two placeholders):

{

"display_name": "spark2-local-compass",

"language": "python",

"argv": [

"<<PYTHON_HOME>>",

"-m",

"IPython.kernel",

"-f",

"{connection_file}"

],

"env": {

"SPARK_HOME": "/opt/spark/",

"PYTHONPATH": "/home/Nimrod/dev/theGarage/:/opt/spark/python/:/opt/spark/python/lib/py4j-<<PY4J_VERSION>>-src.zip",

"PYTHONSTARTUP": "/opt/spark/python/pyspark/shell.py",

"PYSPARK_SUBMIT_ARGS": "--master local[*] --driver-memory 3g --executor-memory 2g pyspark-shell",

"PYSPARK_PYTHON": "<<PYTHON_HOME>>"

}

}

Now, you should be able to observe the new kernel listed in jupyter kernelspec list or in the jupyter UI under the new notebook types.

Example of The new kernel in the Jupyter UI

The current problem with the above is that using the --master local[*] argument is working with Derby as the local DB, this results in a situation that you can’t open multiple notebooks under the same directory.

For most users theses is not a really big issue, but since we started to work with the Data science Cookiecutter the logical structure of the file system puts all the notebooks under the same directory. This will cause an issue every time we want to work simultaneously on multiple notebooks.

I’ve looked for a very long time on finding a solution and finally after a very long time Amit Wolfenfeld found the solution quite fast. The first step is to install postgresql and make sure it runs !

In order to allow pySpark to use postgresql we need the JDBC drivers, download them from here and save them into /opt/spark/jars/.

Next, change the user to the postgres ( sudo su postgres ) and run psql :

CREATE USER hive;

ALTER ROLE hive WITH PASSWORD 'mypassword';

CREATE DATABASE hive_metastore;

GRANT ALL PRIVILEGES ON DATABASE hive_metastore TO hive;

\q

After you ran the command, make sure to restart the postgreql service. The last step is to create a file under the config directory in Spark (Assuming you followed my suggested paths above, the command should be: vim /opt/spark/conf/hive-site.xml

Add the following content to the file:

<configuration>

<property>

<name>javax.jdo.option.ConnectionURL</name>

<value>jdbc:postgresql://localhost:5432/hive_metastore</value>

</property> <property>

<name>javax.jdo.option.ConnectionDriverName</name>

<value>org.postgresql.Driver</value>

</property> <property>

<name>javax.jdo.option.ConnectionUserName</name>

<value>hive</value>

</property> <property>

<name>javax.jdo.option.ConnectionPassword</name>

<value>mypassword</value>

</property> </configuration>

That’s it ! now you can start as many pyspark notebooks you may want.