Setting up Jupyterlab over Spark
By: Shobha Deepthi V
Tags: Jupyter Notebook, Jupyter, Big Data Platform, Jupyterlab, Apache Spark

This post shows how jupyterlab can be setup over an existing spark cluster. To know how to setup a Spark cluster please refer to my earlier post on the same.

Operating System : Red Hat Enterprise Linux Server release 7.6 (Maipo)

Hadoop Version : 3.2.1

Java JRE : 1.8

Python : 3.7

Spark : 3.1.1

Steps

Create a new conda environment for JupyterLab

#conda create -n <Environment Name> python=3.7 jupyterlab pandas scikit-learn seaborn

conda create -n jupHub python=3.7 jupyterlab pandas scikit-learn seaborn

2. Switch to new environment

conda activate jupHub

3. Install needed modules

conda install ipykernel conda install configurable-http-proxy

4. Check the available kernel specifications for Jupyter

jupyter kernelspec list

$ jupyter kernelspec list Available kernels: nlg /users/sparkadm/.local/share/jupyter/kernels/nlg pyspark /users/sparkadm/.local/share/jupyter/kernels/pyspark python3 /auto/csap/conda/envs/jupHub/share/jupyter/kernels/python3 $

5. You would be editing the last file under /auto/csap/conda/envs/jupHub/share/jupyter/kernels/python3. You will add necessay environment varibales, spark submit arguments, jar dependencies etc in this file. This is where you tell Jupyter notebook created from this kernel to submit job to a Spark cluster

{ "argv": [ "/auto/csap/conda/envs/jupHub/bin/python", "-m", "ipykernel_launcher", "-f", "{connection_file}" ], "display_name": "Python 3.7", "language": "python", "env": { "HADOOP_CONF_DIR": "/home/hadoop/conf", "HADOOP_USER_NAME": "sparkadm", "HADOOP_CONF_LIB_NATIVE_DIR": "/home/hadoop/lib/native", "YARN_CONF_DIR": "/home/hadoop/conf", "LD_LIBRARY_PATH":"/auto/csap/conda/envs/jupHub/lib:/usr/lib64/mysql:/usr/local/lib64:/auto/csap/conda/envs/jupHub/:/usr/cisco/packages/oracle/current/lib:/usr/lib64:/usr/local/lib", "PYTHONPATH": "/auto/csap/conda/envs/jupHub/bin:/auto/csap/conda/envs/jupHub/lib/:/home/spark/python:/home/spark/python/lib/py4j-0.10.9-src.zip", "SPARK_HOME": "/home/spark", "PYTHONSTARTUP": "/home/spark/python/pyspark/shell.py", "PYSPARK_SUBMIT_ARGS": "--master spark://171.70.184.87:7077 pyspark-shell" } }

6. Start Jupyterlab

jupyter-lab — no-browser

$ jupyter-lab --no-browser [I 23:18:51.655 LabApp] JupyterLab extension loaded from /auto/csap/python3/lib/python3.6/site-packages/jupyterlab [I 23:18:51.655 LabApp] JupyterLab application directory is /auto/csap/conda/envs/jupHub/share/jupyter/lab [I 23:18:51.661 LabApp] Serving notebooks from local directory: /auto/csap/csap/Jupyter/Dev [I 23:18:51.661 LabApp] Jupyter Notebook 6.4.0 is running at: [I 23:18:51.661 LabApp] http://xyz.abc.com:8888/ [I 23:18:51.661 LabApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

or you can start it as a background deamon

nohup jupyter-lab --no-browser > jupyterlab.log &

7. Jupyterlab is now running on port 8888. If you cannot access it you may want to ensure that port is enabled. As root you need to run below command

iptables -I INPUT -p tcp --dport 8888 -j ACCEPT

8. There are few more settings that can be added such as configuring directory where the notebooks are created in etc. Firstly create the config if does not exist

jupyter notebook --generate-config

9. This will create jupyterhub_config.py file under <USER_HOME_DIRECTORY>/.jupyter folder.

10. You can mention the default notebooks directory here. There are few more settings in this file which could be altered for LDAP connection, Jupyterlab user id and passwords etc.