A step towards industrialization: parameters your code with pyspark and argparse | by Alexandre Wrg | Towards Data Science

argparse is a python library that allows us to write our own command lines to include flexibility in our code. I personally use it in many of my scripts to make my data pipelines more flexible and to form models that are on a moving time window for example. We’ll see some use cases after a quick look at the library.
First, we need to import the library.
import argparse
then we define a “Parser”
parser = argparse.ArgumentParser()
The ArgumentParser object will hold all the information necessary to parse the command line into Python data types. Then we can pass some argument to our parser object.
Let’s say that one :
parser.add_argument(    '--train_start',    help='''Our algorithm deal with a moving window so we need to flexibilise our filename and so on it will be initialised with a flexible name like my_training_set_at_{train_start}''',    required=True,)
we added an argument to our parser, called “train_start”, and created a help message that defines the purpose and use of the argument or whatever else you want to write. Finally, we set up an order condition that forces the use of this argument, if False, it can be ignored.
We will copy the script in a .py file :
Here we define an ArgumentParser object that will store our argument. Then we define two arguments, one that has a value and another that will add them together, let’s detail.
ArgumentParser.add_argument(name or flags...[, action][, nargs][, const][, default][, type][, choices][, required][, help][, metavar][, dest])
we first add an argument with :
a name or flags : ‘integers’
a type : the datatype here is an integer
an help : a message that help the user of the command
a metavar : changes the displayed name.
nargs : which define the number of command-line arguments that should be consumed. The operation'+'. Just like '*', all command-line args present are gathered into a list. Additionally, an error message will be generated if there wasn’t at least one command-line argument present.
In the second added argument there’s is
a name or flags : ‘- -sum’
a default action: which is to return the max if nothing is specified
const : the action of summed the list of integers
a dest : The name of the attribute to be added to the object returned by parse_args()
a helper message
Now, we will save it as “parser_exemple.py” then open a console and look at it :
then try it
Fine for the “tutorial”, let’s see a real exemple now.
We will start with one of my previous articles on pandas-udf and prophet for the prediction and parameterization of this script, which will allow us to use argparse to parameterize our script through a real example and avoid a long useless article. Below is the link of the article with most of the code we will parameterize
https://towardsdatascience.com/pyspark-forecasting-with-pandas-udf-and-fb-prophet-e9d70f86d802
In fact, it’s not very difficult, we will define our parser with several arguments in a different file (this is my personal bias: in fact, you can do everything in the same space).
Thus, we will define a function that will create our ArgumentParser, add the desired argument to it and return the analyzed argument.
In main we import our function and retrieve argument which we put in line when we launched our spark submit.
then launch it through the shell as a python job or spark submit job with the right argument and that’s it.
These basic elements are essential for building a highly versatile pipeline and do not require special development skills.
It may seem simplistic, but this kind of tool is absolutely essential to have flexible pipelines and to make life easier. I use it very often in my projects, especially with airflow in order to program recurring forecasts over time — this will be the subject of a future article.
Thank you for your attention, you will find all the essential points and gists on my github : https://github.com/AlexWarembourg/