Building package for machine learning project in Python | by Mateusz Bednarski | Towards Data Science

Most of machine learning projects I’ve seen are in permanent experiment state with no clear plan for success from a technical perspective. Creating an algorithm that actually solves a viable problem, does it well is difficult by itself. However, managing highly-experimental codebase in a reliable way and making your project ready to deploy is another difficult problem. Especially if you do not have a developer’s background.
In my next series I’ll cover some topics I found useful to make a transition from experiment to production easier. Most of them also help to keep your code reliable and results easier to reproduce.
Here is the list:
Building python package
Creating command line interface for it
Managing dependencies (python and non-python)
Defining dependency graph
Creating automated (unit) tests
Serving it as rest api
Dockerizing
Today I’ll cover creating a python package and command line interface. Also at the beginning, I’ll show two additional key principles that touch third and fourth bullet.
We start with very simple and dirty “prototype”. This is not even an app, just bunch of scripts.
We have empty readme, github-generated license file and gitignore, some bash script and three python files. As data is the most precious resource for data scientist with start with it.
Data files are usually too big to store in code repository and needs to be hosted somewhere else. In this application we use public aclImdb_v1 dataset for sentiment analysis.
Rule #1 Always make sure you can acquire the data once again — and your team know how to do it.
Our poor prototype is already quite good in this. Let’s take a look into download_data.sh
set -emkdir -p data/rawwget http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz -O data/raw/imdb.tar.gztar -xzf data/raw/imdb.tar.gz --directory data/rawrm data/raw/imdb.tar.gz
The script creates directory data/raw, downloads dataset, unpack it and remove no longer necessary archive. Ideally I’d prefer to keep data in my S3 account (in case of link expiration) but we can assume this is good enough. However, in reality your data may be a database dump, bunch of excel files that needs to be merged etc. You should create scripts for all data acquisition steps (e.g.: database dump can be scripted) to make sure it is possible to run whole process in an automated way.
There are situations when it is not possible (because various reasons: technical, legal, compliance etc). In such situation, steps required must be clearly documented. New team member should be able to get the data without asking anyone for help. But remember, best documentation is executable documentation ;)
Now, let’s take a look at requirements.txt
numpypandasscikit-learn
List of required packages is very simple but fine for now. It is very important to track all project’s dependencies. Later we will see that simply listing required packages is not enough, but if you are not using even this simple list, please start.
Rule #2 List all your dependencies and create separate environment
For every project create separate conda/virtual environment and list all dependencies. If your project requires some additional non-python dependencies (like database drivers, system-wide packages) list them all explicitly, or if possible create installation scripts for them. Remember, the number of manual steps must be as limited as possible. This also applies to “one-time activities”. I’m sometimes surprised how often those one-timers are executed.
Now we can move forward to python code.
File dataset.py contains simple class Dataset, for reading dataset (surprise). Code is pretty okay, except hardcoded path (we will fix this later)
Next file, train_model.py is a bunch of statements contained in script file. The script does what is supposed to do, but we will need to work on it. At initial version content is following:
Steps are very simple, the model is trained and stored to file as well as used vectorizer. Later, it is evaluated on test data and a simple report is printed. The script works, but as you probably suppose it has nothing in common with good practices. Unfortunately, much “just experiment” code does not exit this stage of “code quality”. We will break this barrier ;)
The used model is very simple, and its accuracy is poor — but it is not a problem. I keep things as simple as possible as it is not series about creating a good classifier ;) In reality, such training script is more complicated, but it is easier to explain things on simpler examples.
For completeness of the example there is one more script — ask.py. Its role is to provide an interface to make predictions for new data.
We walked through initial state of the “data science experiment”. Now we are going to transform it into production-ready solution.
First step is to create a package. There are many befits with basically no cost. Some of them:
Clear project structure
Ability to create source/binary distribution for deployment
Versioning
Automatic dependency checking
Easy creation of extensions
Separation (e.g. you might not want include training code for prediction environment)
And more :)
Rule #3 Package your code
Creating a package is easy. For first we need to choose name ;) Let’s call it mlp (machine-learning-production). Create such directory, and move source files inside:
mkdir mlpmv dataset.py train_model.py ask.py mlp/
Next, we need to do some slight alignments. Package code should not be executable directly. Therefore it must be contained in classes or functions. By the way, we will extract hardcoded paths into parameters. For dataset we need only to modify class constructor:
Two other files share the same model. It means, that this part should be extracted. Something like this:
Dataset and model are two parts that we were able to extract as reusable components. Two remaining (ask and training scripts) are more like launchers. For now, let’s remove them. We will create a better version of them ;)
There are two more things to do for packaging. First is to create __init__.py in mlp directory with the following content:
from mlp.dataset import Datasetfrom mlp.model import DumbModel
__init__.py is executed at the moment of loading module (you can think about it like “package constructor”). It should not do too many things, but importing parts considered as public part of the package is a very common scenario. This is not required, but allows us to write:
from mlp import Dataset, DumbModel
Instead of
from mlp.dataset import Datasetfrom mlp.model import DumbModel
The last thing is to create setup.py in the project root directory. This file contains all meta-information about the package. It can be very large, but for our purposes (at this moment) a few lines are sufficient:
We just need to define name (this is what you type in pip install XXXX), list of packages (this is what you use in import statement), and version — refer PEP 440.
The project currently looks like this:
Very last thing to do is to install our package. To do this type following (in project root):
pip install -e .
There is no magic: instead of package name we provide a dot which means “install current directory”. The -e switch makes it installed in development mode. This basically means that every change in package source will be immediately reflected in installed version — there would be no need to reinstall it after making changes. You can see your package after executing pip list:
$ pip listPackage         Version    Location                     --------------- ---------- -----------------------------cloudpickle     0.6.1      cycler          0.10.0...mlp             0.0.1.dev1 $HOME/blog/mlp....
To make sure everything is working you can execute python repl and try to instantiate dataset and model.
$ python>>> from mlp import Dataset, DumbModel>>> d = Dataset()>>> m = DumbModel()
Command line interface
We reached halfway point. The second topic for today is to create an interface. Initially, there were two dirty scripts. Instead, I’ll show you how to make a nice command line interface.
Rule #4 Well-define interface for your project. Bunch of scripts is not enough.
There will be two commands. One for training the model, second for asking for predictions. Let’s start by creating file called mlp.py in project root. This will be our “launcher”. Our commands are simple enough to create just two plain functions for them:
Of course, when such function is long it is always a good idea to include it into the package, and just call it from launcher.
Okay, but how will we invoke them? Something like this could be useful:
python mlp.py train data/raw/aclImdb/ model.pickle --vocab-size=5000
Or for prediction:
python mlp.py ask model.pickle "This movie is awesome"
How to create it? There are several possibilities e.g.: argparse or click. But I’m going to use docopt. Install it and do not forget to include it in requirements.txt
With docopt we define the inteface by writtig... help message. May seem crazy ;)
Add following to the beginning of mlp.py:
By reading this docstring you already know what commands are available, what args are required and so one. Docopt will parse this, and create argument parser for us. I encourage you to take a look at docopt’s documentation for more info.
Executing our commands is very simple. Add following to the end of mlp.py:
We just check which command was selected and pass it’s arguments. Simple as that.
Today we will do just one more little thing. Currently our launcher is outside package. Let’s integrate it more tightly. Move our script to mlp directory and rename it to cli.py (command-line-interface). Slightly align docopt help message to:
We will execute our project by mlp-clicommand. I chose not to use plain mlp because it would conflict with mlp directory.
Next update setup.py to:
We added an entry point to the package. It is a console script with executable name mlp-cli. This executable will launch function main from package mlp.cmd
As we edited setup script, there is a need to reinstall package:
pip install -e .
Remember: when you change package code there is no need to reinstall it (as long as you used -e switch. If you change setup.py content you have to reinstall the package.
And now we can type mlp-cli -h into the terminal to verify it works ;)
Yay, we have a single entry point for communicating with our project ;) No more messy scripts!
That was quite a lot. We learned how to create a basic python package and create a command line interface for it. Also, we managed to make it fully integrated with it. Not to forget two basic rules about reproducibility. Right now project structure should look like this:
You can download the code at this moment from project repository; https://github.com/artofai/mlp. Look for tag part1-finished.
In next post I’ll tackle another problem from listed in introduction.
If you like you can do a simple exercise: add an option limit to our app. Dataset is already prepared to handle such parameter ;)
Stay tuned and see you soon :)