Apache Spark on Windows: A Docker approach | by Israel Siqueira | Towards Data Science

Recently I was allocated to a project where the entire customer database is in Apache Spark / Hadoop. As a standard in all my projects, I first went to prepare the development environment on the corporate laptop, which comes with Windows as standard OS. As many already know, preparing a development environment on a Windows laptop can sometimes be painful and if the laptop is a corporate one it can be even more painful (due to restrictions imposed by the system administrator, corporate VPN, etc.).
Creating a development environment for Apache Spark / Hadoop is no different. Installing Spark on Windows is extremely complicated. Several dependencies need to be installed (Java SDK, Python, Winutils, Log4j), services need to be configured, and environment variables need to be properly set. Given that, I decided to use Docker as the first option for all my development environments.
Now, Docker is my “one ring / one tool”, (a reference to Lord of the Rings):
“In the Land of Mordor (Windows) where the shadows lie. One ring to rule them all, One ring to find them, One ring to bring them all, and in the darkness bind them; In the Land of Mordor where the shadows lie.” (Tolkien)
If Docker isn’t an option for you, there are several articles to shed light on the subject:
Installing Apache PySpark on Windows 10
Apache Spark Installation on Windows
Getting Started with PySpark on Windows
Why Docker?
There is no need to install any library or application on Windows, only Docker. No need to ask Technical Support for permission to install software and libraries every week. (They will love you, trust me)Windows will always run at maximum potential (without having countless services starting on login)Have different environments for projects, including software versions. Ex: a project can use Apache Spark 2 with Scala and another Apache Spark 3 project with pyspark without any conflict.There are several ready-made images made by the community (postgres, spark, jupyters, etc.), making the development set-up much faster.
There is no need to install any library or application on Windows, only Docker. No need to ask Technical Support for permission to install software and libraries every week. (They will love you, trust me)
Windows will always run at maximum potential (without having countless services starting on login)
Have different environments for projects, including software versions. Ex: a project can use Apache Spark 2 with Scala and another Apache Spark 3 project with pyspark without any conflict.
There are several ready-made images made by the community (postgres, spark, jupyters, etc.), making the development set-up much faster.
These are just some of the advantages of Docker, there are others which you can read more about on the Docker official page.
With all that said, let’s get down to business and set up our Apache Spark environment.
You can follow the start guide to download Docker for Windows and go for instructions to install Docker on your machine. If your Windows is the Home Edition, you can follow Install Docker Desktop on Windows Home instructions.
When the installation finishes you can restart your machine (remember to save this article in favorites to back from restart).
If you run any error at this point or later, check Microsoft Troubleshoot guide.
You can start Docker from the start menu, after a while you will see this icon on the system tray:
you can right-click on the icon and select Dashboard. On the dashboard, you can click on the configurations button (engine icon on the top right). You will see this screen:
One thing I like to do is unselect the option:
Start docker desktop on your login.
This way docker will not start with windows and I can start it only when I need by the start menu. But it’s a personal choice.
Check Docker Installation
First of all, we need to ensure that our docker installation is working properly. Open a Powershell (or a WSL terminal), I strongly recommend the amazing Windows Terminal, a Windows (Unix-like) terminal that has a lot of features that help us as developers (tabs, auto-complete, themes, and other cool features) and type the following:
~$ docker run hello world
If you see something like this:
Your docker installation is ok.
As I said earlier, one of the coolest features of docker relies on the community images. There’s a lot of pre-made images for almost all needs available to download and use with minimum or no configuration. Take some time to explore the Docker Hub, and see by yourself.
The Jupyter developers have been doing an amazing job actively maintaining some images for Data Scientists and Researchers, the project page can be found here. Some of the images are:
jupyter/r-notebook includes popular packages from the R ecosystem.jupyter/scipy-notebook includes popular packages from the scientific Python ecosystem.jupyter/tensorflow-notebook includes popular Python deep learning libraries.jupyter/pyspark-notebook includes Python support for Apache Spark.jupyter/all-spark-notebook includes Python, R, and Scala support for Apache Spark.
jupyter/r-notebook includes popular packages from the R ecosystem.
jupyter/scipy-notebook includes popular packages from the scientific Python ecosystem.
jupyter/tensorflow-notebook includes popular Python deep learning libraries.
jupyter/pyspark-notebook includes Python support for Apache Spark.
jupyter/all-spark-notebook includes Python, R, and Scala support for Apache Spark.
and many others.
For our Apache Spark environment, we choose the jupyter/pyspark-notebook, as we don’t need the R and Scala support.
To create a new container you can go to a terminal and type the following:
~$ docker run -p 8888:8888 -e JUPYTER_ENABLE_LAB=yes --name pyspark jupyter/pyspark-notebook
This command pulls the jupyter/pyspark-notebook image from Docker Hub if it is not already present on the localhost.
It then starts a container with name=pyspark running a Jupyter Notebook server and exposes the server on host port 8888.
You may instruct the start script to customize the container environment before launching the notebook server. You do so by passing arguments (-e flag) to the docker run command. The list of all available variables can be found in docker-stacks docs.
The server logs appear in the terminal and include a URL to the notebook server. You can navigate to the URL, create a new python notebook and paste the following code:
#create a spark session
from pyspark.sql import SparkSession
spark = SparkSession.builder.master("local[*]").appName("spark_on_docker").getOrCreate()

#Download a README.md file from docker-stacks to test Spark
!wget -q https://github.com/jupyter/docker-stacks/blob/d990a62010aededcda836196c4b04efece7f838f/README.md 

#read the file and count how many rows have the word Jupyter
textFile = spark.read.text("README.md")
rows_with_jupyter = textFile.filter(textFile.value.contains("Jupyter")).count()

print("There's %d rows with the word Jupyter."%rows_with_jupyter)

There's 21 rows with the word Jupyter.

#print the session status
spark

SparkSession - in-memory
SparkContext
Spark UI
 

Voilá! We have our Apache Spark environment with minnimum effort. You can open a terminal and install packages using conda or pip and manage your packages and dependecies as you wish. Once you have finished you can press ctrl+C and stop the container.
If you want to start your container and have your data persisted you cannot run the “docker run” command again, this will create a new default container, so what we need to do?
You can type in a terminal:
~$ docker ps -a
this will list all containers available, to start the same container that you create previously, type:
~$ docker start -a pyspark
where -a is a flag that tells docker to bind the console output to the terminal and pyspark is the name of the container. To learn more about docker start options you can visit Docker docs.
In this article, we can see how docker can speed-up the development lifecycle and help us mitigate some of the drawbacks of using Windows as the main OS for development. Microsoft is doing a great job with WSL, Docker, and other tools for developers and engineers even supporting GPU processing through docker and WSL. The future is promising.
Thanks for reading this article, I hope that this small guide can help you and give valuable insights to anyone who decides to use Docker for Windows as your “One ring”.
Please feel free to ask or provide feedback in the comments section.