How to Setup PySpark and MongoDB for Building Data Pipeline
By: Vandany Lubis
Tags: Data Engineering, Data Science, Pyspark, Data Pipeline, Mongodb

OVERVIEW

Have you heard about a data pipeline before? If not let me explain you to a brief definition of a data pipeline. The data pipeline or also can be mention as the ETL pipeline is the set of processes that transform raw data into actionable answers to business questions. Data science pipelines automate the processes of data validation; extract, transform, load (ETL); machine learning and modeling; revision; and output, such as to a data warehouse or visualization platform (Snowflake, n.d.).

Fig. 0 Spark + MongoDB (Source: https://www.mongodb.com/integrations/mongodb-spark)

Okay, now I will show you how to set up a data pipeline architecture using PySpark, and MongoDB.

SETUP AND INSTALLATION

1 . First of all, we need to install a third-party library such as FindSpark, PySpark, and PyMongo. You can install the library using the command pip install library-name, or conda install library-name if you are using Anaconda Prompt.

2. Secondly, we need to install Apache Spark, which you can install from here. After that, make a new folder named “Spark” in the C:// directory and copy-paste the folder inside the zip file you have downloaded to the Spark folder.

Fig.1 Inside the Spark folder

3. After we install the Apache Spark now we need to configure the Spark home environment for running Spark in windows. First, go to This PC->Properties->Advance System Settings->Environment Variables, then click New, create a new variable name as SPARK_HOME the value of the variable is the directory of the Spark file, for example: “C:\Spark\spark-3.0.3-bin-hadoop2.7”. After that, click variable “Path”, create a new variable as %SPARK_HOME%\bin and then click OK.

Fig.2 Set up environment

4. After we configure Spark home, now we need to configure our Hadoop home. The first thing we need to do is download the file “winutils.exe” from here this file is needed to run Hadoop in windows. After you downloaded the file, copy and paste it to the “bin” folder inside the Spark folder. Now back to This PC->Properties->Advance System Settings->Environment Variables click New, create a new variable name as HADOOP_HOME the value of the variable is the same as SPARK_HOME. After that, click variable “Path”, create a new variable as %HADOOP_HOME%\bin and then click OK.

5. Once we have installed Spark and Hadoop next thing we can do now is installing Java 8 JDK from here.

6. After we have installed Spark and Java, now is the time to check whether our Spark is installed and configured properly. Open command prompt as admin then types “spark-shell”. If we see a welcome message, congratulations the Spark is installed successfully in our machine. If not please create a new folder as “tmp” in C:// then make a new folder inside as “hive”, it is optional in case you are using an older version of Spark.

Fig.3 Spark shell

7. After the Spark is running successfully the next thing we need to do is download MongoDB, and choose a community server. In this project, I am using MongoDB 5.0.2 for Windows. When you are installing MongoDB you will encounter a server customization setup, for personal use just make it default do not change anything. After installing MongoDB, open the installation directory then open Mongo.exe, then type “db” if the output is “test” it means the MongoDB was installed successfully.

Fig.4 Testing MongoDB

8. As long as we know to interface with MongoDB all we can do is using the command line which is not really user-friendly hence we need a tool to handle this situation. The software we will use is NoSQLBooster but if you prefer to use the command line is no problem. Once we already NoSQLBooster installed the next thing we need to do is establish the connection, open NoSQLBooster->File->Connect->Create->Default settings->Test Connection. After the connection is running successfully we need to create a database under the localhost. You have to right-click on the localhost then create a database. You can name your database whatever you want, in this case, I named my database “Quake” for the next project.

Fig.5 NoSQLBooster connection setup

After MongoDB is completely installed, the next thing we can do is testing the PySpark with the Jupyter Notebook. You can simply do this:

Fig.6 Testing PySpark with Jupyter Notebook

If you see the “hello spark” message, you can verify that PySpark is working successfully with the Jupyter Notebook.

Okay, once our data pipeline architecture is completely installed and the PySpark is integrated with the Jupyter Notebook now we can do our data science project.

I hope my article helps you to complete your journey to become a professional data scientist.

Thank you!

Prepared by: Vandany Lubis

BIG THANKS TO EBISYS TO TEACH ME HOW TO TO THIS