Shell Scripts for Data Science in Python | by Louis de Bruijn | Towards Data Science

Let’s learn a little bit of shell scripting, as it will greatly improve your ability to supply and parse command-line arguments (CLA) to your Python scripts.
A shell script is a computer program designed to be run by the Unix shell, a command-line interpreter. Typical operations performed by shell scripts include file manipulation and program execution, and this is precisely why it is relevant to learn the basics as a data scientist.
Why shell scripting?
Single-dash CLA with getopts in shell scripts
Double-dash CLA in shell scripts
Python’s built-in argparse
argparse lesson for boolean values: bool and store_true
Putting it all together
Shell scripts enable you to automate a collection of command-line operations in a single script, executing them line by line. Your project is ready for these automation steps, and it is handy to learn shell sooner than later.
A few examples where I used it in my work as a data scientist: virtual environment management, instantiation of PySpark tables, execution of Python linters, updating and building static documentation websites, execution of production pipelines with specific arguments, etc.
Bash scripts have a built-in option getoptsto deal with both boolean flags and flags that have optional arguments. getopts can only deal with single-dash CLA, such as -a. It reads the :ab: in the while-statement as follows
: the first colon enables silent error reporting mode: getopts will not produce any diagnostic messages for missing option arguments or invalid options. By suppressing automatic error messages, you can handle errors with custom messages.
a case represents a boolean CLA -a.
b: case represents an optional argument CLA -b some_value.
while getopts :ab: flag
do
  case "${flag}" in
    a) first_val=a;;
    b) second_val=${OPTARG};;
    :)  echo 'missing argument!'
        exit 1 ;;
    \?) echo 'invalid option!'
        exit 1 ;;
  esac
done
Inside the while-loop, we can catch our command-line arguments with the <character>) syntax; we can then assign the CLA to a variable such as first_variable. There are two special cases
: case above catches missing argument errors. This is only relevant if all arguments are mandatory.
/? case represents any invalid option errors. Note that ? needs to be escaped or quoted to match a literal ? as it otherwise matches any single character like with regular expressions.
We can call our script with the following command-line arguments
$ sh run.sh -a -b hello
Double-dash arguments are a lot more difficult than it seems because there is no built-in method to work with double-dashes: we have to iterate over each CLA and match arguments. This is fairly simple for boolean arguments such as --help.
for argument in "$@"
do
  if [[ $argument == "--help" ]] || [[ $argument == "-h" ]]; then
    echo "usage: $ sh run.sh --help"
    exit
  fi
done
We use a for loop iteration over all the command line arguments captured in $@. If-elif-else statements are closed with fi. This particular one looks for two string comparisons captured in a double-pipe || OR-statement for each argument: --help and -h.
I make use of bash-supported double brackets [[ ... ]], but you can also use single brackets. Read more on the difference between the two here.
If we want to supply optional arguments such as --arg1 some_var, it becomes a lot more complex, and I would refer to this documentation of a custom-built arg parser in bash.
Also good to know, a single, double dash -- is used in most Bash built-in commands and many other commands to signify the end of command options, after which only positional arguments are accepted, such as $ sh run.sh -a one -b two -- three four.
Now on to more familiar territory: Python scripts and its built-in argparse. Using this built-in argument parser adds a lot of functionalities out of the box. argparse uses all data types, enables in-place conversion, default values, and defines single — and double—- dash CLA. Our Python script run.py looks as follows.
import argparse
import datetime as dt
​
parser = argparse.ArgumentParser()
​
parser.add_argument("--name", nargs="?", default="")
parser.add_argument("--age", nargs="?", type=int, default=1)
parser.add_argument("--date", nargs='?', default=dt.date.today().strftime('%Y%m%d'))
parser.add_argument("-d", "--debug", nargs="?", type=bool, default=False)
parser.add_argument('--alive', action="store_true")
​
args = parser.parse_args()
​
print(args)
The arguments are stored via the command parser.parse_args() in an object that we retrieve via each argument’s name, such as args.age or args.debug. If we print args without supplying any arguments, we get the following object with default values:
Namespace(name='', age=1, date='20210709', debug=False, alive=False)
One lesson I learned the difficult way is that boolean values can be supplied in two ways, either via an argument with type=bool or via an argument with action="store_true". Whichever way you choose differentiates its behavior vastly. Let’s look at the following two cases:
$ python3 run.py --name Louis --age 27 --alive --debug>>> Namespace(name='Louis', age=27, date='20210709', debug=None, alive=True)$ python3 run.py --name Louis --age 27 --alive --debug True>>> Namespace(name='Louis', age=27, date='20210709', debug=True, alive=True)
If you supply type=bool, you need to add a flag True or False for the argument to not be None. You can only circumvent this by adding a default value in the case of no value supplied. Make no mistake, though that if you only supply the flag --debug without a boolean value, it will result in None.
On the other hand, if you apply action=store_true, you do not need to supply any boolean flag: supplying the argument --alive stores it as True and not supplying it stores is as False. Vice-versa with action=store_false. More on this in Python’s argparse documentation here.
Let’s put our knowledge to the test and create a shell script run.sh that supplies command-line arguments provided to our run.py. I’ve added some useful functionalities in our shell script to explain its usage and a help function and avoid misuse of the arguments.
The -z ${var_name} syntax returns True if the variable is null (None). ${var_name} is how you call a variable in Bash or shell and "var_name = $var_name” in a string. If we run run.sh and print the arguments in run.py, we will see the following usages:
$ sh run.sh -n Elvis -b 19350108 -a 42>>> Namespace(name='Elvis', age=42, date='19350108', debug=False, alive=False)$ sh run.sh -n Ringo -b 19400707 -a 81 -d False -l>>> Namespace(name='Ringo', age=81, date='19400707', debug=False, alive=True)
I hope you enjoyed this story and happy coding!