How to setup the Python and Spark environment for development, with good software engineering practices
By: Bogdan Cojocar
Tags: Python, Apache Spark

In this article we will discuss about how to set up our development environment in order to create good quality python code and how to automate some of the tedious tasks to speed up deployments.

We will go over the following steps:

setup our dependencies in a isolated virtual environment with pipenv

how to setup a project structure for multiple jobs

how to run a pyspark job

how to use a Makefile to automate development steps

to automate development steps how to test the quality of our code using flake8

how to run unit tests for PySpark apps using pytest-spark

running a test coverage, to see if we have created enough unit tests using pytest-cov

Step 1: setup a virtual environment

A virtual environment helps us to isolate the dependencies for a specific application from the overall dependencies of the system. This is great because we will not get into dependencies issues with the existing libraries, and it’s easier to install or uninstall them on a separate system, say a docker container or a server. For this task we will use pipenv.

To install it on a mac os system for example run:

brew install pipenv

To declare our dependencies (libraries) for the app we need to create a Pipfile in the route path of our project:



url = '

verify_ssl = true

name = 'pypi' [[source]]url = ' https://pypi.python.org/simple' verify_ssl = truename = 'pypi' [requires]

python_version = "3.6" [packages]

flake8 = "*"

pytest-spark = ">=0.4.4"

pyspark = ">=2.4.0"

pytest-cov = "*"

There are three components here. In the [[source]] tag we declare the url from where all the packages are downloaded, in [requires] we define the python version, and finally in [packages] the dependencies that we need. We can bound a dependency to a certain version, or just take the latest one using the “*”symbol.

To create the virtual environment and to activate it, we need to run two commands in the terminal:

pipenv --three install

pipenv shell

Once this is done once, you should see you are in a new venv by having the name of the project appearing in the terminal at the command line (by default the env is takes the name of the project):

(pyspark-project-template) host:project$

Now you can move in and out using two commands.

Deactivate env and move back to the standard env:

deactivate

Activate the virtual environment again (you need to be in the root of the project):

source `pipenv --venv`/bin/activate

Step 2: the project structure

The project can have the following structure:

pyspark-project-template

src/

jobs/

pi/

__init__.py

resources/

args.json

word_count/

__init__.py

resources/

args.json

word_count.csv

main.py

test/

jobs/

pi/

test_pi.py

word_count/

test_word_count.py

Some __init__.py files are excluded to make things simpler, but you can find the link on github to the complete project at the end of the tutorial.

We basically have the source code and the tests. Each job is separated into a folder, and each job has a resource folder where we add the extra files and configurations that that job needs.

In this tutorial I have used two classic examples — pi, to generate the pi number up to a number of decimals, and word count, to count the number of words in a csv file.

Step 3: running a job using spark-submit

Let’s see first how the main.py files looks like:

When we run our job we need two command line arguments: — job, is the name of the job we want to run (in out case pi or word_count) and — res-path, is the relative path to the jobs. We need the second argument because spark needs to know the full path to our resources. In a production environment, where we deploy our code on a cluster, we would move our resources to HDFS or S3, and we would use that path instead.

Before explaining the code further, we need to mention that we have to zip the job folder and pass it to the spark-submit statement. Assuming we are in the root of the project:

cd src/

zip -r ../jobs.zip jobs/

This will make the code available as a module in our app. Basically in main.py at line 16, we are programatically importing the job module.

Both our jobs, pi and word_count, have a run function, so we just need to run this function, to start the job (line 17 in main.py). We also pass the configurations of the job there.

Let’s have a look at our word_count job to understand further the example:

This code is defined in the __init__.py file in the word_count folder. We can see here that we use two config parameters to read the csv file: the relative path, and the location of the csv file, in the resources folder. The rest of the code just counts the words, so we will not go into further details here. It’s worth to mention that each job has, in the resources folder an args.json file. Here we actually define the configuration that we pass to the job. This is the config file of the word_count job:

{

"words_file_path": "/word_count/resources/word_count.csv"

}

So we have all the details now to run our spark-submit command:

spark-submit --py-files jobs.zip src/main.py --job word_count --res-path /your/path/pyspark-project-template/src/jobs

To run the other job, pi, we just need to change the argument of the — job flag.

Step 4: writing unit tests, and running them with coverage

To wrote tests for pyspark application we use pytest-spark, a really easy to use module.

The word_count job unit tests:

We need to import the functions that we want to test from the src module. The more interesting part here is how we do the test_word_count_run. We can see there is no spark session initialised, we just received it as a parameter in our test. This is thanks to the pytest-spark module, so we can concentrate on writing the tests, instead of writing boilerplate code.

Next let’s discuss about code coverage. How do we know if we write enough unit tests? Easy, we run a test coverage tool, that tells us what code is not tested yet. For python we can use the pytest-cov module. To run all the tests using code coverage we have to run:

pytest --cov=src test/jobs/

where — cov flag is telling pytest where to check for coverage.

Test coverage results:

---------- coverage: platform darwin, python 3.7.2-final-0 -----------

Name Stmts Miss Cover

-----------------------------------------------------

src/__init__.py 0 0 100%

src/jobs/__init__.py 0 0 100%

src/jobs/pi/__init__.py 11 0 100%

src/jobs/word_count/__init__.py 9 0 100%

-----------------------------------------------------

TOTAL 20 0 100%

Our test coverage is 100%, but wait a minute, one file is missing! Why is main.py not listed there?

If we consider that we have python code that we don’t need to test, we can exclude it from the reports. To do this we need to create a .coveragerc file in the root of our project. For this example it looks something like this:

[run]

omit = src/main.py

Step 5: running static code analysis

Great, we have some code, we can run it, we have unit tests with good coverage. We are done right? Not yet! We also need to make sure that we write easy to read code, following python best practices. To do this we have to inspect our code with a python module called flake8.

To run it:

flake8 ./src

It will analyse the src folder. If we have clean code, we should get no warnings. But no, we have a few issues:

flake8 ./src

./src/jobs/pi/__init__.py:13:1: E302 expected 2 blank lines, found 1

./src/jobs/pi/__init__.py:15:73: E231 missing whitespace after ','

./src/jobs/pi/__init__.py:15:80: E501 line too long (113 > 79 characters)

Let’s see the code:

We can see we have an E302 warning at line 13. That means we need an extra line between the two methods. Then an E231 and E501 at line 15. The first warning on this line, tells us that we need an extra space between the range(1, number_of_steps +1), and config[ , and the second warning notifies us that the line is too long, and it’s hard to read (we can’t even see it in full in the gist!).

After we solve all the warnings the code definitely looks easier to read:

Step 6: putting it all together with a Makefile

Because we have run a bunch of commands in the terminal, in this final step we are looking into how to simplify and automate this task.

We can create a Makefile in the root of the project as the one bellow:

.DEFAULT_GOAL := run init:

pipenv --three install

pipenv shell analyze:

flake8 ./src run_tests:

pytest --cov=src test/jobs/ run:

find . -name '__pycache__' | xargs rm -rf

rm -f jobs.zip cd src/ && zip -r ../jobs.zip jobs/ spark-submit --py-files jobs.zip src/main.py --job $(JOB_NAME) --res-path $(CONF_PATH)

If we want to run the tests with coverage, we can simply type:

make run_tests

And if we want to run the pi job:

make run JOB_NAME=pi CONF_PATH=/your/path/pyspark-project-template/src/jobs

That’s all folks! I hope you find this useful.

As always the code is stored on github.