Data Science in Production — Advanced Python Best Practices
By: Victor Kostyuk
Tags: Software Engineering, Python, Data Science, Product Development, Clean Code

Ten years ago, any quantitative PhD with a few statistics courses under their belt and an inkling of writing code could call themselves a data scientist — and not be entirely wrong. I should know — I was one of them. But as the field of data science has evolved, this is no longer the case. It’s not enough to be able to put together a few models in a notebook. Today, data scientists work iteratively in teams to build stable, scalable, and extendable data science products which drive core revenue-generating activities for their companies. The following guide fills a gap in the existing literature by focusing on data science software engineering practices required to build effective data products.

As an introduction, I suggest reading Andrew Fowler’s excellent post that outlines three key data science best practices: command line executability, YAML configuration files, and model testing.

In this guide, I extend this foundation to a full suite of Python data science best practices:

1) Configure effectively

Box up your config

Make all parameters configurable

Ensure portability by using environment variables

2) Use non-code to improve development

Write docstrings before coding

Use typehints and mypy

Keep code and documentation together

Make it replicable using virtual environments

3) Manage complexity

Decouple, decouple, decouple

Organize project directory structure

Use a single entry point

Create a walking skeleton

4) Minimize bugs

Log, don’t print

Unit test, a lot

Assert to validate data

Track provenance

1) Configure effectively

Box up your config

Save time and sanity by accessing hierarchical YAML config files using Box. This package allows for dot-based access to YAML parameters, rather than Python’s traditional dictionary syntax; e.g., you will write cfg.base.path.data instead of cfg["base"]["path"]["data"].

Here’s an example YAML config.

Pip install python-box and Box up this config.

Then, in the rest of your project, use “.” notation for easy access:

Best practice: analogously to defaultdict , there’s a defaultbox . Here’s an idiomatic way to use it with config files to facilitate reuse and modularity of functions/methods.

The above function looks for n_estimators in params . If it is not present, it checks cfg.base.rf_params.n_estimators , and if that’s missing as well, sets a default value (200).

Make all parameters configurable

Making all parameters in your package configurable has three advantages:

You can quickly adjust the ETL, model, and other pipeline components by changing a YAML file rather than editing code. Your collaborators and users can make substantial modifications without changing (or even understanding) the underlying code. Your package can be readily integrated in automated pipelines that dynamically modify configuration parameters before launching your package.

For example, here’s a section of the config defining training parameters:

Now we can use importlib to dynamically import and instantiate the specified sampler and classifier models.

Then, the Sampler and Classifier can be changed simply by modifying the config file, without changing any code.

Ensure portability by using environment variables

In the above config file, we have different DB paths depending on the environment: dev or prod. Use environment variables to determine where you are, and parse the config accordingly. For example, let’s say the variable ENVIRONMENT is either dev or prod , depending where your package is run.

This has the additional advantage of removing the need for .base (and .dev or .prod ) when accessing config parameters. Now you can just call cfg.path.data instead of cfg.base.path.data .

Here are three related best practices: (1) save your environment variables to the config (line 10 above) so that you can easily access them through the config without having to import os each time; (2) never save plaintext passwords or credentials in the config or in your code— use environment variables for that; and (3) save the environment variables in a .env file (add it to .gitignore ) and load them using dotenv.

2) Use non-code to improve development

Write docstrings before coding

Poor design is the original sin from which subsequent difficulties in extending, modifying, and understanding your program arise. Learn clean design — it’s one of the highest-value time investments for a data scientist.

Plan the design and align with teammates by writing the docstrings for your key modules, classes, and functions prior to coding them. These can be reviewed (PR them!) and revised with the full team, so that everyone provides input and gains understanding of the whole package architecture, not just the pieces they are responsible for coding.

Of course a lot will change as you and your team develop. Nevertheless, initial planning will set you on a smoother development and refactoring trajectory. As Eisenhower said, “plans are useless, but planning is essential.”

Use typehints and mypy

Unlike statically typed languages, such as Java and Scala, Python does not require type declarations. Since Python 3.6, however, they can be added to your code using typehints.

Typehints serve two purposes: documentation and static checking using mypy. Using typehints obviates the need for documenting types inside docstrings. Additionally, the mypy package reads typehints and, if you run it prior to execution, it’ll reveal elusive type errors that crop up intermittently and are hard to detect without static checking.

Think of mypy as a fast, “free” way to detect many issues (or at the very least, inconsistent typehints) without having to perform manual testing. Some IDEs, including Pycharm and VS Code, also check typehints statically and highlight inconsistencies as you code.

Keep code and documentation together

Keep documentation version-controlled together with code. For docs to be useful, they must be updated together with the code they’re documenting. Write documentation in markdown and render as clean, navigable HTML using Sphinx¹.

And, remember those comprehensive, Google-style formatted docstrings you’ve been assiduously writing before coding? Add them automatically to the documentation using Sphinx autodoc.

Make it replicable using virtual environments

Managing dependencies while developing multiple packages in Python can be a chore. Ensure you and your collaborators have the same development environment by using virtual environments, which encapsulate and isolate both the Python interpreter and dependencies of your package. Two common virtual environment options are conda and venv. Pick one —it doesn’t matter which. I prefer venv, since it’s built into Python 3.3 and above.

Create the virtual environment using venv and install dependencies as follows:

Above, we create an environment with Python 3.6 interpreter, add three dependencies (more precisely, at least 3 — they each may have other dependencies pip will install), and then write the list of current dependencies into a requirements.txt file. This file, and not the env directory itself is added to git. Then your collaborators can recreate the same environment by initializing a new virtual environment as in line 2, activating it as in line 3, and executing pip install -r requirements.txt in order to install the required dependencies.

Using conda is analogous, except that instead of requirements.txt , an environment.yml file is normally used to save dependency information. You can find the details in the official documentation.

3) Manage complexity

Decouple, decouple, decouple

Modular components are easier to reuse, debug, and extend. Ensure that key parts of your pipeline — including data sourcing, preprocessing, training, evaluation, and prediction — are standalone.

Source: Pipe data from multiple original sources into a staging DB (e.g., for simple applications, sqlite), separate files (e.g., images for training a classifier), or denormalized tables saved as pickle, Parquet, feather, or other formats. Preprocess: Read from staging DB/files, use asserts to validate key staged data properties, engineer features, transform, and save to new DB table(s) or files. Train: Read from files, use asserts to validate key preprocessed data properties, define model and training scheme (e.g., cross-validation), train model, and save (e.g., as a pickle, or h5 for TF/Keras models). Evaluate: Load model, apply to test data, save metrics. Predict: Load model, apply to new data, save predictions.

Each step should be governed by configuration parameters. If only a few configuration parameters are needed (e.g., paths to a few source files and an output file), provide them as arguments to a function wrapped with a click CLI (see Andrew’s post for details). If many parameters are needed, use a separate YAML config, with its path given as an argument to the click CLI wrapped function.

Organize project directory structure

A clean, logical directory structure will reduce the cognitive load involved in finding, understanding, and changing functionality — both for you and for your collaborators and users.

Here’s a directory structure I recommend. This structure is for a project named supermodel. The files are for the most part examples — yours may be different.

supermodel/ Root project directory

├── docs/ Documentation (.md/.rst files)

├── configs/ Config files (.yml)

├── logs/ Logfiles

├── notebooks/ EDA and validation (.ipynb)

├── scripts/ Deployment, Dockerfile, etc.

├── env/ Virtual env, add to .gitignore

├── supermodel/ Top level package dir

│ ├── source/ Data sourcing (.py, not shown)

│ ├── preprocess/ Preprocessing (.py, not shown)

│ ├── model/ Modeling (.py)

│ │ ├── __init__.py Designates as (sub)package

│ │ ├── train_eval.py

│ │ └── predict.py

│ ├── utils/ Util functions used in source,

│ │ ├── __init__.py preprocess, model.

│ │ ├── spark.py

│ │ ├── date_time.py

│ │ └── io.py

│ ├── tests/ Unit tests

│ │ ├── source/

│ │ ├── preprocess/

│ │ ├── model/

│ │ │ ├── __init__.py

│ │ │ ├── test_train_eval.py

│ │ │ └── test_predict.py

│ │ ├── utils/

│ │ │ ├── __init__.py

│ │ │ ├── test_spark.py

│ │ │ ├── test_date_time.py

│ │ │ └── test_io.py

│ │ └── __init__.py

│ ├── __main__.py Package execution entry point:

│ ├── __init__.py "python -m supermodel"

│ ├── config.py Loads and Boxes configuration

│ └── run.py Called from __main__.py

├── README.md Intro to package

├── setup.py Installing the package

├── requirements.txt Lists dependencies

├── .gitignore Files/dirs to git ignore

└── LICENSE.md License

A few things to note:

The root directory is not a Python package (it has no __init__.py ), but the same-named directory supermodel , and its subdirectories, are Python packages. Unit tests (more on those below, in the “Unit test, a lot” best practice) are in supermodel/tests (from the root) and are organized in directories and files mirroring the structure supermodel . For instance, functions of supermodel/utils/spark.py are tested in supermodel/tests/utils/test_spark.py . setup.py contains a call to the setup function from setuptools , with the install_requires loading requirements from requirements.txt , like this. To simplify your life, avoid duplication of requirements in setup.py and requirements.txt . config.py should load and Box your config file(s) and add in any environment variables, as described in the sections above. Then, in all other modules, you would import the config Box as from supermodel.config import cfg . __main__.py calls the main() function in run.py , which is the single entry point for all functionality (see below “Use a single entry point” best practice). It enables the user to execute the package in a standard way: python -m supermodel (with optional arguments following). Do you find it too confusing that the root directory and top-level package directory have the same name? If so, change one of the names (e.g., scikit-learn has root directory “scikit-learn” and top level package “sklearn”. In Pandas, on the other hand, both are named “pandas”). Personally, I avoid multiplicity of names, so I use the same name for both. This structure is not meant to be exhaustive, or set in stone. Depending on the nature of the project and the approach, you may have multiple top-level packages (e.g., a separate, extensive ETL or reporting process), interfaces/ directory for abstract classes if taking an object-oriented approach to the pipeline, a separate directory for integration and end-to-end tests, etc.

Use a single entry point

A key principle of good design is to wrap complex business logic in separate components connected with simple APIs. Having a single entry point is an application of this principle to your whole package: it’ll make it simple for users and collaborators to access package functionality and use it in their workflows².

The entry point should be the package itself: python -m supermodel (to use the name of the example package above). To allow this, create a __main__.py like this:

Then in your run.py you would define two functions: main_cli which is decorated with click to process command line arguments, and main , which is called from main_cli and defines the execution of your package. (Why separate main_cli and main ? The former can’t be imported, due to the click decorations; the latter can.) Here’s an example main from supermodel/run.py:

The imports in lines 3–6 and the tasks dictionary can be refactored using the config file and importlib (see the “Make all parameters configurable” best practice section above). Remember to not include if __name__ == "__main__": in any of the other files in the package. After all, you want __main__.py to be the single entry point to all package functionality.

Create a walking skeleton

Waiting to connect all the components of your pipeline — from source data ingestion to delivering the resulting predictions — to the end of the project is a recipe for disaster. By creating a complete dummy pipeline you will kill two birds with one stone: first, de-risk the external system integration challenges and second, create a “walking skeleton” on top of which you’ll develop the real functionality piece by piece. The walking skeleton will allow you to test that the pipeline executes successfully while you are developing.

What does the initial version of the walking skeleton entail? It varies by pipeline step (see the “Decouple, decouple, decouple” section above):

Source: For each data store from which data is sourced, query a small sample of the data (not saved anywhere). Instead, write a dummy data table/file to the designated staging DB/bucket/directory for preprocess to pick up. Preprocess: Read the dummy table/file written by source. Don’t do anything with it. Write dummy preprocessed data for predict. Predict: Read the dummy table/file from preprocess. Write a dummy prediction in the format you expect the real prediction to have to the destination where it’ll be read by downstream systems (e.g., a Pandas dataframe with the expected columns saved as a csv to the agreed-upon S3 bucket).

Notify downstream users of your model to verify that they can read the dummy predictions and that the format of the predictions is suitable. As you develop, rerun the pipeline to incrementally test each additional piece you implement. In fact, schedule it to run in the same way the final product will typically be run, such as using a daily Airflow DAG run, cron job, or event triggers. By doing so, you’ll also be able to verify that all required external systems — data sources and predictions destination — are reliably accessible over time.

4) Minimize bugs

Log, don’t print

Python provides extensive logging facilities. By logging, instead of using print statements, you can specify (1) a dynamic log format, indicating where in your package the statement originated, and (2) different levels of severity, from debug (lowest) to error (highest). You’ll also be able to flexibly channel your logs to files, standard out and other streams, and even email.

While you don’t need to know Python’s full logging functionality, you should master the basics and use them consistently in your packages.

Configure your logger using a YAML configuration file, like this one.

Then, in your config.py (after you load the cfg YAML), set up the logger based on this configuration:

In each module, set the logger name to the module name in the beginning of the file, following the imports (line 5 below):

Yes, you have to do it in every file. But it’s a small price to pay for clear logs that indicate which module each log message is coming from.

Unit test, a lot

Strive to write pure functions — functions that take arguments, process them, and return the result without changing state³. Look askance at all functions which return None; 90% or more of those should be I/O write functions.

Pure functions are easier to reason about, reuse in your pipeline, and unit test.

Write unit tests for all functions that encode business logic and perform data transformations. Isolate in small functions — and do not unit test — I/O, graphing, and orchestration (i.e., functions or scripts that execute your pipeline in the intended order). Also, do not unit test functions which thinly wrap external (e.g., Pandas or Scikit-learn) functionality — those packages have their own suites of unit tests. If the wrapper contains substantial logic, consider separating it. For example, split a wrapper that sets up model architecture and then trains into two functions: model setup (test this) and model fitting (do not test — sklearn/TF function). The thicker the wrapper, the greater the benefit of testing.

Assert to validate data

Use assert statements to validate necessary assumptions about data and key calculation results: if the assertion is false, your program will halt and display the assertion message. This will save the user time and frustration tracing an error raised later in the program due to, for instance, faulty input data⁴.

Below are examples of three types of assertions:

Assertions of necessary properties of data inputs (line 8) — columns, value constraints, dtypes, etc. Assertions about the environment (lines 11, 12) — environment variables, key package versions, availability of external resources. Assertions about results (lines 19, 20) — prior to returning a result, check that it makes sense.

Track provenance

Especially during rapid, iterative model development, you want to know which preprocessed data, saved model, or results were generated by what code.

There are a few ways to accomplish this:

No versioning: This is a valid option if the modeling project is simple and running the pipeline is very quick (i.e., you can always checkout a previous commit and quickly rerun to generate all outputs). Manual versioning: Backup important “milestone” outputs with descriptive filenames. Git hash versioning: In your saving functions (for saving preprocessed data, models, results), append the git hash of the current commit using GitPython. For example:

4. Full-featured data versioning: Use a system such as DVC.

For many purposes, options 2 and 3 are sufficient.

Sip from the Firehose

The above practices are a lot to take in all at once. The good news is that you don’t have to. Add them one at a time to your repertoire until you develop familiarity and comfort. They will serve you well on your journey from Jupyter-notebook slinging data science cow(boy|girl) to a clean-coding, product-developing professional.

Thanks to Andrew Fowler, Brad Allen, Cloves Almeida, and Akos Furton for providing feedback on an early draft of this guide.

Footnotes:

[1] Markdown is simpler to write than reStructuredText, but it doesn’t capture the full functionality of Sphinx. For more complex documentation, use reStructuredText.

[2] In this guide, I’m using “entry point” in the general sense of a method for executing your package. I’m not referring to the entry_points argument in setup.py. You would not normally use those entry points in a data science project.

[3] Do you need to keep state in an organized way (as opposed to having a bunch of dicts and DataFrames floating around)? Use classes/objects. Within classes, use the @staticmethod decorator to designate utility methods that don’t need to use class/object attributes, and make those methods pure.

[4] Caveat: if you do not have control of how your package is deployed, and it may be deployed by an inexperienced devops engineer who thinks that Python is C++ and decides to use "optimization” flags, then using assertions in your code is dangerous because they are skipped by the optimization. In this case, use conditionals and raise exceptions instead of using assertions. Or you can provide explicit instruction on how to correctly deploy the package. (Explanatory note: unlike in C++, the optimization flags in Python provide close to zero improvement in execution speed. Don’t use them, don’t let friends use them.)