Installing Apache PySpark on Windows 10
By: Uma Gajendragadkar
Tags: Data Sicence, Windows 10, Pyspark, Apache Spark, Installation

Installing Apache PySpark on Windows 10

Over the last few months, I was working on a Data Science project which handles a huge dataset and it became necessary to use the distributed environment provided by Apache PySpark.

I struggled a lot while installing PySpark on Windows 10. So I decided to write this blog to help anyone easily install and use Apache PySpark on a Windows 10 machine.

1. Step 1

PySpark requires Java version 7 or later and Python version 2.6 or later. Let’s first check if they are already installed or install them and make sure that PySpark can work with these two components.

Installing Java

Check if Java version 7 or later is installed on your machine. For this execute following command on Command Prompt.

If Java is installed and configured to work from a Command Prompt, running the above command should print the information about the Java version to the console. Else if you get a message like:

‘java’ is not recognized as an internal or external command, operable program or batch file.

then you have to install java.

a) For this download java from Download Free Java Software

b) Get Windows x64 (such as jre-8u92-windows-x64.exe) unless you are using a 32 bit version of Windows in which case you need to get the Windows x86 Offline version.

c) Run the installer.

d) After the installation is complete, close your current Command Prompt if it was already open, reopen it and check if you can successfully run java --version command.

2. Step 2

Python

Python is used by many other software tools. So it is quite possible that a required version (in our case version 2.6 or later) is already available on your computer. To check if Python is available and find it’s version, open Command Prompt and type the command python --version

If Python is installed and configured to work from Command Prompt, running the above command should print the information about the Python version to the console. For example, I got the following output on my laptop:

C:\Users\uug20>python --version

Python 3.7.3

Instead if you get a message like

“ ‘python’ is not recognized as an internal or external command, operable program or batch file.”

It means you need to install Python. To do so,

a) Go to the Python download page.

b) Click the Latest Python 2 Release link.

c) Download the Windows x86–64 MSI installer file. If you are using a 32 bit version of Windows download the Windows x86 MSI installer file.

d) When you run the installer, on the Customize Python section, make sure that the option Add python.exe to Path is selected. If this option is not selected, some of the PySpark utilities such as pyspark and spark-submit might not work.

e) After the installation is complete, close the Command Prompt if it was already open, reopen it and check if you can successfully run python --version command.

3. Step 3

Installing Apache Spark

a) Go to the Spark download page.

b) Select the latest stable release of Spark.

c) Choose a package type: select a version that is pre-built for the latest version of Hadoop such as Pre-built for Hadoop 2.6.

d) Choose a download type: select Direct Download.

e) Click the link next to Download Spark to download a zipped tar file ending in .tgz extension such as spark-1.6.2-bin-hadoop2.6.tgz.

f) For installing Apache Spark, you don’t have to run any installer. Extract the files from the downloaded tar file in any folder of your choice using the 7Zip tool/ other tools to unarchive.

Make sure that the folder path and the folder name containing Spark files do not contain any spaces.

I created a folder called spark on my D drive and extracted the zipped tar file in a folder called spark-2.4.3-bin-hadoop2.7. So all Spark files are in a folder called D:\spark\spark-2.4.3-bin-hadoop2.7. Let us refer to this folder as SPARK_HOME in this post.

To test if your installation was successful, open Command Prompt, change to SPARK_HOME directory and type bin\pyspark. This should start the PySpark shell which can be used to interactively work with Spark.

The last message provides a hint on how to work with Spark in the PySpark shell using the sc or sqlContext names. For example, typing sc.version in the shell should print the version of Spark. You can exit from the PySpark shell in the same way you exit from any Python shell — by typing exit().

The PySpark shell outputs a few messages on exit. So you need to hit enter to get back to the Command Prompt.

4. Step 4

Configuring the Spark Installation

Initially when you start the PySpark shell, it produces a lot of messages of type INFO, ERROR and WARN. Let us see how to remove these messages.

Spark installation on Windows does not include the winutils.exe utility by default which is used by Spark. If you do not tell your Spark installation where to look for winutils.exe, you will see error messages when running the PySpark shell such as

“ ERROR Shell: Failed to locate the winutils binary in the hadoop binary path java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.”

This error message does not prevent the PySpark shell from starting. However if you try to run a standalone Python script using the bin\spark-submit utility, you will get an error. For example, try running the wordcount.py script from the examples folder in the Command Prompt when you are in the SPARK_HOME directory.

“bin\spark-submit examples\src\main\python\wordcount.py README.md”

Installing winutils

Let’s download the winutils.exe and configure our Spark installation to find winutils.exe.

a) Create a hadoop\bin folder inside the SPARK_HOME folder.

b) Download the winutils.exe for the version of hadoop against which your Spark installation was built for. In my case the hadoop version was 2.6.0. So I downloaded the winutils.exe for hadoop 2.6.0 and copied it to the hadoop\bin folder in the SPARK_HOME folder.

c) Create a system environment variable in Windows called SPARK_HOME that points to the SPARK_HOME folder path.

d) Create another system environment variable in Windows called HADOOP_HOME that points to the hadoop folder inside the SPARK_HOME folder.

Since the hadoop folder is inside the SPARK_HOME folder, it is better to create HADOOP_HOME environment variable using a value of %SPARK_HOME%\hadoop. That way you don’t have to change HADOOP_HOME if SPARK_HOME is updated.

If you now run the bin\pyspark script from a Windows Command Prompt, the error messages related to winutils.exe should be gone.

5. Step 5

Configuring the log level for Spark

There are still a lot of extra INFO messages in the console every time you start or exit from a PySpark shell or run the spark-submit utility. So let’s make one more change to our Spark installation so only warning and error messages are written to the console. In order to do this:

a) Copy the log4j.properties.template file in the SPARK_HOME\conf folder as log4j.propertiesfile in the SPARK_HOME\conf folder.

b) Set the log4j.rootCategory property value to WARN, console.

c) Save the log4j.properties file.

Now any informative messages will not be logged to the console.

Summary

In order to work with PySpark, start Command Prompt and change into your SPARK_HOME directory.

a) To start a PySpark shell, run the bin\pyspark utility. Once your are in the PySpark shell use the sc and sqlContext names and type exit() to return back to the Command Prompt.

b) To run a standalone Python script, run the bin\spark-submit utility and specify the path of your Python script as well as any arguments your Python script needs in the Command Prompt. For example, to run the wordcount.py script from examples directory in your SPARK_HOME folder, you can run the following command:

“ bin\spark-submit examples\src\main\python\wordcount.py README.md “

6. Step 6

Important : I was facing a problem in installation

I was getting following error message after completing the installation procedure on my Windows 10 machine.

File “C:\Users\uug20\Anaconda3\lib\site-packages\zmq\backend\cython\__init__.py”, line 6, in <module> from . import (constants, error, message, context, ImportError: DLL load failed: The specified module could not be found.

Solution:

I just figured out how to fix it!

In my case I wasn’t aware that I have to add THREE miniconda related paths to PATH environment variable.

C:\ Users\uug20\Anaconda3

C:\ Users\uug20\Anaconda3\Scripts

C:\ Users\uug20\Anaconda3\Library\bin

After that I did not get any error messages and pyspark started working correctly and opened Jupyter notebook after typing pyspark in command prompt.

Hope it works for you too!