How to Install easily Spark for Python | by Papa Moryba Kouate | Towards Data Science

Introduction
When we work with Big Data, we need more computational power that we can get with a distributed system of multiple computers. Moreover, to work effectively into the big data ecosystem, we also need a cluster computing framework which permits us to perform processing tasks quickly on large data sets.
The two most famous cluster computing frameworks are Hadoop and Spark that are available for free as open-source.
If we want to compare Apache Spark with Hadoop, we can say that Spark is 100 times faster in memory and 10 times faster on disk.
For what concerns on-perm installations Hadoop requires more memory on disk and Spark requires more RAM, that means setting up a cluster could be very expensive.
Today we can solve that problem with the services of cloud computing provided by AWS and Azure. If you are interested in knowing something more about them, in particular a topic like the Cloud Data Warehouse, let me suggest to you my article that you can find here.
Instead, in this article, I will show you how to install the Spark Python API, called Pyspark. Installing Pyspark on Windows 10 requires some different steps to follow and sometimes we can forget these steps. So, with this article, I hope to give you a useful guide to install Pyspark with no problems.
Part I: Check your Java version and download Apache Spark
I assume that you have on your PC a Python version at least 3.7.
So, to run Spark, the first thing we need to install is Java. It is recommended to have Java 8 or Java 1.8.
So, open your Command Prompt and control the version of your Java with the command that you can see below. If you have an old version, you can download it here.
Java -version
out:
Now, we have to download Spark that you can easily find here. The following frame show you the steps that you will see when you are in the site.
Download Apache Spark1. Choose a Spark release: 3.0.0(Jun 18 2020)--selected2. Choose a package type: Pre-built for Apache Hadoop 2.7 --selected3. Download Spark: spark-3.0.0-bin-hadoop2.7.tgz
Above you can observe how I set my version of Spark. I selected the recent Spark release and the Pre-built package for Apache Hadoop 2.7.
Then in the 3rd step, we can download our Spark version by clicking on the link that will open another web page in which you have to click on the first downloading version suggested.
When the download is finished, extract the compressed file in a folder called spark. Remember that the path of this folder will be used in the next part.
Part II: Download winutils.exe and Set Up your Environment
The second step is to download winutils.exe that you can find here. In my case, I chose the version hadoop-3.0.0, and I downloaded it. You can create a folder called winutils and put it there.
Now, it is time to set up our environment. The first thing to do is to go in the windows search bar and digit “edit the system environment variables”.
Go and click on the Environment Variables.
When you will be in the group of the System Variables, create three variables called HADOOP_HOME, SPARK_HOME and JAVA_HOME.
In HADOOP_HOME →put the path of the location wintulis folder created before.
In SPARK_HOME →put the path of the location of spark folder that you created before.
In JAVA_HOME →put the path of the location of your JAVA program.
In the following images you can see how you can set up your variables.
Then in the variable Path as you can see below, we can add the following two paths:
%SPARK_HOME%\bin
%JAVA_HOME%\bin
Part III: Run it on your Anaconda and fly to Jupyter
Now you can open the Anaconda Prompt and change the directory into SPARK_HOME directory and type bin\pyspark.
out:
So, you are ready to fly to Jupyter and try Pyspark with the following code. If there are no errors that means your Pyspark has been installed rightly.
import findsparkfindspark.init()
Conclusion
Now you have your Pyspark and you can start to learn and practice how it works by making some data manipulation. If you have some trouble to install it, feel free to contact me.
I send out a periodical newsletter. If you would like to join please sign up via this link.
In addition to my newsletter, we can also get in touch in my telegram group Data Science for Beginners.