Successful spark-submits for Python projects. | by Kyle Jarvis | Towards Data Science

TL; DR: The best way to learn is to do, and the best way to learn something hard is to do it in small steps. This is a guide for Python practitioners (probably data scientists) working on a package in an isolated development environment, who want to start successfully executing Spark jobs using that package on a cluster. The big data tech stack can seem daunting at first: this article is aimed at those who are just getting to grips with working effectively with a cluster, who want to get their project off the ground and running, then feel confident to explore, build on this experience, and refine their style.
Enough, already! Take me to the guide (scroll down).
The problem.
When writing, developing and testing our Python packages for Spark, it’s quite likely that we’ll be working in some kind of isolated development environment; on a desktop, or dedicated cloud-computing resource. Crucially, the Python environment we’ve been at liberty to put together, the one with our favourite minor versions of all the best packages, is likely to be different from the Python environment(s) accessible to a vanilla spark-submit job executed on our friendly neighbourhood cluster. Generally, it’s desirable to have some degree of flexibility, and be able to quickly incorporate additional dependencies into a prototype environment. In the heat and excitement of developing our code, the foresight to prepare a cluster-wide environment compatible with our package, ready to receive and execute our Spark job may have eluded us. Now, this unforeseen hurdle is stopping us from running our project at scale, which is sad for a data scientist.
A solution.
In what follows we’ll walk through the steps necessary to get a Python package running on Spark using spark-submit, in a transparent and accessible way. The steps that are covered in the guide are:
How to package and prepare our Python projects for successful execution on a Spark cluster.Preparing a minimal working environment with the version of Python that’s right for our project, and upload this to Hadoop, making it accessible to all the workers at execution time.Package up our project dependencies.Preparing the spark-submit script to bring all the above together, in an intelligible way.
How to package and prepare our Python projects for successful execution on a Spark cluster.
Preparing a minimal working environment with the version of Python that’s right for our project, and upload this to Hadoop, making it accessible to all the workers at execution time.
Package up our project dependencies.
Preparing the spark-submit script to bring all the above together, in an intelligible way.
Target audience.
The existing resources relevant to this topic (see further reading) are mostly aimed at producing production-grade Spark jobs with Python. They are really great, but could pose a steep learning curve to those less familiar with the intricacies and nuances of working Spark in cluster mode; there’s a lot to take in. If you’re just starting out with these kinds of big data technologies, you’ll want a good understanding of the basics. This guide focuses on the basics and key concepts, highlighting the things you’ll definitely encounter when starting out.
Preamble.
By trade, I’m a data scientist. I work on a commodity cluster running Hadoop and use Spark to build big-data pipelines for machine-learning and data-science projects, processing extremely large data sets containing billions of rows. Development happens in an isolated environment, with Anaconda as the Python distribution. Typically, I’ll have set up a couple of different virtual environments in order to decouple the Spark processing steps from other aspects of the project.
In development, my project will be structured something like this.
my_data_science_project├───data├───models├───envs│   ├───/spark_env/│   └───/ml_env/├───conf├───notebooks│   ├───model_explore.ipynb│   └───eda.ipynb└───src    ├───data_pipeline    │   ├───functions.py    │   ├───processing.py    │   └───__init__.py    ├───app.py    └───__init__.py
I’ll use a Jupyter notebook to play around and tweak transformations, scrutinising the results until they match what I’m trying to achieve. Eventually, once all the code is doing exactly what it should be, the next step is to package it up, and call the transformation steps from a script that we will pass to a spark-submit job. The spark-submit job will setup and configure Spark as per our instructions, execute the program we pass to it, then cleanly release the resources that were being used. A simply Python program passed to spark-submit might look like this:
"""spark_submit_example.pyAn example of the kind of script we might want to run. The modules and functions of our package can be imported and accessed in the usual way. Command line arguments can be accessed in parsed in the usual way. We can even read in files in the usual way. In fact, its all very... usual..."""import yamlimport sysimport src.data_pipeline.processingfrom src.app import get_sparkdef run(ss, appConfig):    # Do some steps using the appConfig    input = ss.read.parquet(appConfig['input'])    output = src.data_pipeline.processing.process(input)    output.write.parquet(appConfig['output'], mode = 'overwrite')if __name__ == "__main__":    #Get some configuration settings    args = sys.argv    appConfig = args[1]    with open(appConfig) as file_handler:        appConfig = yaml.load(file_handler, Loader=yaml.Loader)    # Suppose the Spark session builder is wrapped in this function.    ss = get_spark()    run(ss, appConfig)
Getting this to work smoothly on a cluster, fresh out of an isolated development environment can be tricky. There are lots of moving parts. This guide exists because, I’ve gone through this process myself, and I think having this to refer to would have sped things up considerably.
Disclaimer: I won’t begin to go into how to properly configure and create a Python object bound to a Spark session. That is a topic in its own right, and it’s assumed that you’ve had success in doing that already. Here we delegate the intricacies of this task to the hypothetical get_spark() .
1. Packaging the project.
The first step is to package up all the python files, modules and scripts that belong to the package, i.e. the contents of the./src/ directory. There’s nothing special about this step, you can read about the use of setuptools here. Create your setup.py file and python setup.py bdist_egg . After this step, we’ll have some new files and directories, including a.egg , which for the purposes of this guide is just like a .zip . When we run our spark-submit job later, we’ll let the program know about the egg file containing our project source code, and it will send this to the driver and workers. When our application is launched, the top-level of any artefacts (eggs, zips) passed to --py-files is added to PYTHON_PATH , which is how we can import our package from a program passed to spark-submit.
my_data_science_project├++ setup.py      ]- python setup.py bdist_egg to produce the below.├++ dist/                      │   └++ my_data_science_project-0.1-py2.7.egg└++ build/                     
2. Preparing a minimal working environment.
When running my Spark jobs with a Python driver program, I like to use a minimal working environment. It’s too easy to end up with a bloated virtual environment when prototyping and developing, so this is a useful point to revisit whether any pruning can be done. If the program we’re running is ultimately expressed as pure Spark (vs. doing anything with numpy, pandas) then I find I can get very far with only the builtin modules and a yaml parser.
Let’s go ahead and use conda to create a new environment without any of the packages that are installed by default (which we probably don’t need for our spark job).
conda create \--name spark_submit_env \--no-default-packages \--copy \--prefix ./envs \python=xx 
Running the above command will give us an extra directory in ./envs/ that contains our minimal working environment. We are likely to use this environment over and over again as we tweak our Spark program, so we’ll upload it to the Hadoop file system, rather than shipping and distributing it every time we want to run a Spark job. We’ll upload our environment to Hadoop as a .zip, that will keep everything neat, and we can tell spark-submit that we’ve created an archive we’d like our executors to have access to using the --archives flag. To do this, first follow these steps:
cd ./envs/spark_submit_env/zip -r ..spark_submit_env.zip .
cd ./envs/spark_submit_env/
zip -r ..spark_submit_env.zip .
Now move up one level and run the following command:
hadoop fs -copyFromLocal ./spark_submit_env.zip /hdfs/env/path/
Remember this hdfs path, we’ll refer to it later in our spark-submit script. The commands ran above will have made the below additions to our project directory.
my_data_science_project├───envs│   ├───spark_env│   ├───ml_env│   ├++ spark_submit_env      ]- zip this up│   └++ spark_submit_env.zip  ]- upload to hdfs├───conf
3. Packing up the dependencies.
Having prepared and uploaded our spark_submit_env to hdfs, we now turn to preparing the specific dependencies that our Spark job depends upon. There are multiple ways of achieving the same result here, and it’s pretty straightforward. These steps will be intuitive for anyone who has ever packaged a project, however, we do want to make sure we have the right versions of the packages for the specific version of Python we have in our spark_submit_env . Scribble down the packages we need for our Spark job in a requirements.txt :
requirements.txtpyyamlanother_package_you_love~
Then, source activate spark_submit_env , so that which pip reassuringly tells us /path/to/your/project/envs/bin/pip . Now we can be confident the packages we need for our spark-submit job will play nicely with the environment we’ve prepared. We can now pip install the dependencies listed in requirements to a specified directory that we’ll zip up and pass to Spark.
pip install -t ./dependencies -r requirements.txt
my_data_science_project    ├++ requirements.txt     ]- pip install this├++ requirements/        ]- to here, then zip it up├++ requirements.zip     ]- like this.
Note that we could directly package the dependencies as part of the working environment setup over the course of development and distribute this, but explicitly listing and bundling the package dependencies separately keeps things nice and transparent. We can also cut out superfluous packages that are useful for eda, visualisation etc., but not directly required for the steps delegated to Spark.
4. The spark-submit script.
This is where we bring together all the steps that we’ve been through so far. This is the script we will run to invoke Spark, and where we’ll make it aware of the Python environment we prepared, our packaged project and its dependencies. The spark-submit script will look something like this:
#!/bin/bashPYTHON_ZIP="hdfs:////hdfs/env/path/spark_submit_env.zip#pythonlib"PYSPARK_PYTHON="./pythonlib/bin/python2.7"PROJECT_EGG=./dist/my_data_science_project-0.1-py2.7.eggSPARK_CMD = "spark-submit \--conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=${PYSPARK_PYTHON} \--archives ${PYTHON27_ZIP} \--py-files ${PROJECT_EGG},./dependencies.zip \--master yarn \--deploy-mode cluster \--files ./conf/appConf.yml#appConf.yml \spark_submit_program.py appConf.yml arg2 ..."eval ${SPARK_CMD}
Let’s break this down.
PYTHON_ZIP="hdfs:////hdfs/env/path/spark_submit_env.zip#pythonlib"
Here we declare the variable PYTHON_ZIP and assign to it the file url where we uploaded our Python environment in step 2. The #pythonlib suffix is a little trick that allows us to specify name aliases when running Spark on YARN. We can use it to specify a file name alias for anything passed to --py-files or --archives . This alias is the actual name the Spark app will see. You can read more here.
--conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=${PYSPARK_PYTHON}
Here we’re setting the environment variable PYSPARK_PYTHON to point to the python executable that we package into our spark_submit_env. Search for the above here to read more.
--archives ${PYTHON_ZIP}
Anything that’s passed to --archives get’s unzipped into the working directory of each executor, so passing the hdfs url of the zipped environment is how these executors access the binaries and modules of the environment we prepared.
--files ./conf/appConf.yml#appConf.yml
Individual files passed to --files are also copied to the executors working directories, again we can name them using the /path/to/file#alias syntax. For example, we can pass a yaml file to be parsed by the driver program, as illustrated in spark_submit_example.py .
spark_submit_example.py appConf.yml arg2 arg3 ...
After specifying our [OPTIONS] we pass the actual Python file that’s executed by the driver:spark_submit_example.py, as well as any command line arguments for the program, which can be parsed in the usual way. For example, the first argument might be a yaml file appConf.yml , and there might be subsequent arguments to provide more context for the program.
RUN RUN RUN.
We’re finally in a position to execute the spark-submit script that we’ve described above:
./spark_submit_script.sh
One final word of caution, when working with virtual environments, be cautious of modifications to PATH . I’ve run into issues previously when trying to spark-submit without remembering to deactivate a particular virtual environment.
If you’ve made it this far I hope it’s because you’ve found the contents of this guide helpful — I’d love to hear if that’s the case. I’d be more than willing to expand on any specifics, should anyone be sufficiently engaged to request that. I’m very open to feedback. So, if you spot an inconsistency or mistake, please shout; I’ll promptly remove the article and delete all traces of it ever having existed, and my account, and my LinkedIn. I will leave the inhospitable environments pictogram though, that took me a long time.
When you’re confident with the steps that have been covered in this guide, you might like to look at the following resources:
How to setup the Python and Spark environment for development, with good software engineering practices
Building Production PySpark Jobs
Best Practices Writing Production-Grade PySpark Jobs