Apache SPARK using Jupyter in LINUX : Installation and Setup
By: Karthikeyan Mohanraj
Tags: Machine Learning, Big Data, Pyspark, Apache Spark, Analytics

We are living in a world where everything seems to be moving fast. How often do we say or hear, “oh my gosh it’s time-up already, what I did?”. So, we need to save our time and be productive, thus reducing work on installation process of Pyspark environment here illustrated the processes flow in easier.

Let’s focus on installation of Pyspark using Jupyter in LINUX (Ubuntu) environment. First and foremost for users from Windows and Mac, to install VirtualBox (https://www.virtualbox.org/wiki/Downloads) and install OS Ubuntu18.04 (https://ubuntu.com/download/desktop) in VirtualBox.

Let’s open a Terminal in Ubuntu environment and check in-built python version in Ubuntu,

$ python3

Now, install Jupyter Notebook in the same environment, provide sudo password as ubuntu credential for below installation,

$ sudo apt install python3-pip

$ sudo apt install python3-notebook jupyter jupyter-core python-ipykernel

Use this below command and open Jupyter to check it’s working,

$ jupyter notebook

Let’s update all currently installed packages with below command,

$ sudo apt-get update

Now, we need Java package to be installed for using Spark environment,

$ sudo apt-get install default-jre

$ java -version

Again, will setup Scala package installation using below command,

$ sudo apt-get install scala

$ scala -version

Requires to install ‘Py4J’, it enables Python programs running in a Python interpreter to dynamically access Java objects in a Java Virtual Machine,

$ pip3 install py4j

Now, it’s time to play with Big Data environment setup. Yes, Apache Spark!! Lets, download the latest available version of Apache Spark (https://spark.apache.org/downloads.html) and keep it in ‘/home’ directory of Ubuntu,

Spark Download version and file in highlighted

Will extract the Spark downloaded file which is located in ‘/home’ folder,

$ sudo tar -zxvf spark-2.4.5-bin-hadoop2.7.tgz

Now, will create a environment and organize all setup, installation into a same working clusters,

$ export SPARK_HOME=’home/ubuntu/spark-2.4.5-bin-hadoop2.7'

$ export PATH=$SPARK_HOME:$PATH

$ export PYTHONPATH=$SPARK_HOME/python:$PYTHONPATH

Need to setup driver environment for Jupyter Notebook to perform in SPARK environment to basically use pyspark package,

$ export PYSPARK_DRIVER_PYTHON=”jupyter”

$ export PYSPARK_DRIVER_PYTHON_OPTS=”notebook”

$ export PYSPARK_PYTHON=python3

If you would have noticed your Spark folder in home directory, it will be remains with lock symbol. Whereas, need to allow the operations permission access the spark environment. Hence, to change the file permission and making the file readable, writable and executable will be using below code,

$ sudo chmod 777 spark-2.4.5-bin-hadoop2.7

Now, the SPARK setup is ready. Let’s check the pyspark import function from spark environment through terminal,

$ cd spark-2.4.5-bin-hadoop2.7/python/

$ python3

and import pyspark to verify it’s working,

Python Setup in Linux Environment

Now, let’s get to know how to access Jupyter Notebook from SPARK environment, before that needs to enable the access permission to use Jupyter Notebook through SPARK. Set back directory, follow below commands,

$ cd

$ cd spark-2.4.5-bin-hadoop2.7/

$ sudo chmod 777 python

$ cd python

$ sudo chmod 777 pyspark

$ jupyter notebook

Jupyter Notebook Environment to check pyspark

That’s it!! Yes, you have started using Pyspark from Linux environment. This is easiest process to setup Pyspark environment using Linux terminal commands without any other additional cases to change. In case, if you face any issues in the above setup feel free to leave your comments or also you can reach me through my Linkedin profile (https://www.linkedin.com/in/karthikeyan-mohanraj-814929171/).

Happy Analyzing !!!