How to connect Jupyter Notebook to remote spark clusters and run spark jobs every day? | by Teng Peng | Towards Data Science

As a data scientist, you are developing notebooks that process large data that does not fit in your laptop using Spark. What would you do? This is not a trivial problem.
Let’s start with the most naive solution without install anything on your laptop.
“No notebook”: SSH into the remote clusters and use Spark shell on the remote cluster.“Local notebook”: downsample the data and pull the data to your laptop.
“No notebook”: SSH into the remote clusters and use Spark shell on the remote cluster.
“Local notebook”: downsample the data and pull the data to your laptop.
The problem of “No notebook” is the developer experience is unacceptable on Spark shell:
You cannot easily change the code and get the result printed like what you have in Jupyter notebook or Zeppelin notebook.It is hard to show images/charts from Shell.It is painful to do version control by git on a remote machine because you have to set up from the very beginning and make git operations like git diff.
You cannot easily change the code and get the result printed like what you have in Jupyter notebook or Zeppelin notebook.
It is hard to show images/charts from Shell.
It is painful to do version control by git on a remote machine because you have to set up from the very beginning and make git operations like git diff.
The second option “Local notebook”: You have to downsample the data and pull the data to your laptop (downsample: if you have 100GB data on your clusters, you downsample the data to 1GB without losing too much important information). Then you could process the data on your local Jupyter notebook.
it creates a few new painful problems:
You have to write extra code to downsample the data.Downsampling could lose vital information about the data, especially when you are working on visualization or machine learning models.You have to spend extra hours to make sure your code for original data. If not, it takes extra extra hours to figure out what’s wrong.You have to guarantee the local development environment is the same as the remote cluster. If not, it is error-prone and it may cause data issues that are hard to detect.
You have to write extra code to downsample the data.
Downsampling could lose vital information about the data, especially when you are working on visualization or machine learning models.
You have to spend extra hours to make sure your code for original data. If not, it takes extra extra hours to figure out what’s wrong.
You have to guarantee the local development environment is the same as the remote cluster. If not, it is error-prone and it may cause data issues that are hard to detect.
Ok, “No notebook” and “Local notebook” are obviously not the best approach. What if your data team has access to the cloud, e.g. AWS? Yes, AWS provides Jupyter notebook on their EMR clusters and SageMaker. The notebook server is accessed through AWS Web console and it is ready to use when the clusters are ready.
This approach is called “Remote notebook on a cloud”.
The problems of “remote notebook on the cloud” are
You have to set up your development environment every time the clusters get to spin up.If you want your notebook run on different clusters or regions, you have to manually & repeatedly get it done.If the clusters are terminated unexpectedly, you lost your work on those clusters.
You have to set up your development environment every time the clusters get to spin up.
If you want your notebook run on different clusters or regions, you have to manually & repeatedly get it done.
If the clusters are terminated unexpectedly, you lost your work on those clusters.
This approach, ironically, is the most popular one among the data scientists who have access to AWS. This can be explained by the principle of least effort: It provides one-click access to remote clusters so that data scientists can focus on their machine learning models, visualization, and business impact without spending too much time on clusters.
Besides “No notebook”, “Local notebook”, and “Remote notebook on Cloud”, there are options that point spark on a laptop to remote spark clusters. The code is submitted via a local notebook and send to a remote spark cluster. This approach is called “Bridge local & remote spark”.
You can use set the remote master when you create sparkSession
val spark = SparkSession.builder().appName(“SparkSample”).master(“spark://123.456.789:7077”).getOrCreate()
The problems are
you have to figure out how to authenticate your laptop to remote spark clusters.it only works when Spark is deployed as Standalone not YARN. If your spark cluster is deployed on YARN, then you have to copy the configuration files/etc/hadoop/conf on remote clusters to your laptop and restart your local spark, assuming you have already figured out how to install Spark on your laptop.
you have to figure out how to authenticate your laptop to remote spark clusters.
it only works when Spark is deployed as Standalone not YARN. If your spark cluster is deployed on YARN, then you have to copy the configuration files/etc/hadoop/conf on remote clusters to your laptop and restart your local spark, assuming you have already figured out how to install Spark on your laptop.
If you have multiple spark clusters, then you have to switch back and forth by copy configuration files. If the clusters are ephemeral on Cloud, then it easily becomes a nightmare.
“Bridge local & remote spark” does not work for most of the data scientists. Luckily, we can switch back our attention to Jupyter notebook. There is a Jupyter notebook kernel called “Sparkmagic” which can send your code to a remote cluster with the assumption that Livy is installed on the remote spark clusters. This assumption is met for all cloud providers and it is not hard to install on in-house spark clusters with the help of Apache Ambari.
It seems “Sparkmagic” is the best solution at this point but why it is not the most popular one. There are 2 reasons:
Many data scientists have not heard of “Sparkmagic”.There are installation, connection, and authentication issues that are hard for data scientists to fix.
Many data scientists have not heard of “Sparkmagic”.
There are installation, connection, and authentication issues that are hard for data scientists to fix.
To solve problem 2, sparkmagic introduces Docker containers that are ready to use. Docker container, indeed, has solved some of the issues in installation, but it also introduces new problems for data scientists:
Designed for shipping applications, the learning curve of docker container is not considered friendly for data scientists.It is not designed to used intuitively for data scientists who come from diverse technical backgrounds.
Designed for shipping applications, the learning curve of docker container is not considered friendly for data scientists.
It is not designed to used intuitively for data scientists who come from diverse technical backgrounds.
The discussion of docker containers will stop here and another article that explains how to make Docker containers actually work for data scientists will be published in a few days.
To summarize, we have two categories of solutions:
Notebook & notebook kernel: “No notebook”, “Local notebook”, “Remote notebook on cloud”, “Sparkmagic”Spark itself: “Bridge local & remote spark”.
Notebook & notebook kernel: “No notebook”, “Local notebook”, “Remote notebook on cloud”, “Sparkmagic”
Spark itself: “Bridge local & remote spark”.
Despite installation and connection issues, “Sparkmagic” is the recommended solution. However, there are often other unsolved issues that reduce productivity and hurt developer experience:
What if other languages, python and R, are required to run on clusters?What if the notebook is going to be run everyday? What if the notebook is going to be run only if another notebook run succeed?
What if other languages, python and R, are required to run on clusters?
What if the notebook is going to be run everyday? What if the notebook is going to be run only if another notebook run succeed?
Let’s go over the current solutions:
Set up a remote Jupyter server and SSH tunneling (Reference). This definitely works but it takes time to set it up, and notebooks are on the remote servers.Set up a cron scheduler. Most data scientists are OK with cron scheduler, but what if the notebook failed to run? Yes, a shell script can help but are the most majority of data scientists comfortable writing shell script? Even if the answer is yes, data scientists have to 1. get access to finished notebook 2. to get a status update. Even if there are data scientists that are happy with writing shell scripts, why should every data scientist write their own scripts to automate the exact same stuff?Set up Airflow. This is a very popular solution among data engineers and it can get stuff done. If there are Airflow servers supported by data engineers or data platform engineers, data scientists can manage to learn the operators of Airflow and get it to work for Jupyter Notebook.Set up Kubeflow and other Kubernetes-based solutions. Admittedly kubeflow can get stuff done, but in reality how many data scientists have access to Kubernetes clusters, including managed solutions running on the cloud?
Set up a remote Jupyter server and SSH tunneling (Reference). This definitely works but it takes time to set it up, and notebooks are on the remote servers.
Set up a cron scheduler. Most data scientists are OK with cron scheduler, but what if the notebook failed to run? Yes, a shell script can help but are the most majority of data scientists comfortable writing shell script? Even if the answer is yes, data scientists have to 1. get access to finished notebook 2. to get a status update. Even if there are data scientists that are happy with writing shell scripts, why should every data scientist write their own scripts to automate the exact same stuff?
Set up Airflow. This is a very popular solution among data engineers and it can get stuff done. If there are Airflow servers supported by data engineers or data platform engineers, data scientists can manage to learn the operators of Airflow and get it to work for Jupyter Notebook.
Set up Kubeflow and other Kubernetes-based solutions. Admittedly kubeflow can get stuff done, but in reality how many data scientists have access to Kubernetes clusters, including managed solutions running on the cloud?
Let’s reframe the problems:
How to develop on the local laptop with access to remote clusters?How to operate on the remote clusters?
How to develop on the local laptop with access to remote clusters?
How to operate on the remote clusters?
The solutions implemented by Bayesnote can get everything done in 5 minutes. Bayesnote is a notebook orchestration platform for Jupyter notebook:
Orchestrate notebooks on different clusters within 5 minutes
One-button access to remote Jupyter notebook
Run Jupyter notebooks with dependency and retry
View output notebook with one-button
There is
No learning curve. There is no abstraction to learn, no docker/Kubernetes knowledge is required.
No operation cost. There is no extra database to install, no ports to open. It can install itself into clusters without hassles.
The tutorial can be found https://github.com/Bayesnote/Bayesnote