Design Patterns for Machine Learning
Tags: Machine Learning, Efficiency, Design Patterns, Standardization, Methodology

With Machine learning all over the place, it is becoming increasingly important to capture best practices and solutions to tackle common ML problems. Design patterns are a way of capturing these problems and providing reusable answers using generic and well-proven ML designs. They are ways of thinking when designing solutions or building ML systems.

Now the question is , Can we abstract best machine learning practices into design patterns?

As per Wikipedia, “Design patterns are formalized best practices that the programmer can use to solve common problems when designing an application or system”.

Here’s a new book published recently on the topic, “Machine Learning Design Patterns”. The book introduces 30 design patterns in machine learning in detail which are structured into categories such as Problem representation, Model Training, Resilient Serving, Reproducibility and Responsible AI.

In this post, I would like to focus on summarizing few design patterns from the book that I found interesting.

1. Rebalancing: Problem representation Design pattern

Imbalanced datasets are a common scenario in classification problems such as Fraud Detection, Anomaly Detection, Spam Detection, etc. But the typically ML models used for classification work with the assumption that all the classes are balanced and thus resulting in poor predictive performance. Some of the most common strategies used to handle imbalanced datasets are given as follows:

1. Choosing the right performance metric: Though accuracy is the most commonly used metric, it might not be the best one to use in case of imbalanced datasets as the class distribution is skewed. The goal in this case is to maximize both precision and recall, and hence metrics such as F1 score or AUC are more efficient to evaluate the performance.

2. Sampling methods: Resampling is a widely adopted technique to balance the samples within the classes in the dataset.

a) Over-sampling: This approach involves oversampling minority class. The simple approach involves duplicating the samples from the minority class which doesn’t add any additional information to the dataset. The other advanced techniques like SMOTE (Synthetic Minority Oversampling Technique) synthesize new samples from the existing data.

b) Down-sampling: This approach involves choosing a random subset of the majority class to reach the same size of the minority class, which might cause loss of information.

3. Weighted classes: This approach involves using penalized learning algorithms which increase the cost of misclassifications in the minority class. That is, the cost of classification mistakes in the minority class is proportional to how under-represented the minority class is.

2. Transform : Reproducibility Design Pattern

The key idea of this design pattern is to separate input from features. To train a model, we extract features from raw input. But in most of the machine learning problems, inputs are not directly used as features. A lot of transformations such as standardization, scaling, encoding, etc. are applied on the input to convert them into features that are fed into the algorithms. But if these transformations are carried out as preprocessing steps, it’s a problem to reproduce them at the prediction time as the input for prediction will still be in the raw format. So, its important to clearly separate out the inputs from the features, encapsulate the preprocessing steps and include it into the model to ensure reproducibility. This design principle allows an easy migration of a machine learning model to production by separating inputs, features and transformations. TensorFlow provides tf.transform to carry out transformations efficiently and saving them, so that they can be reproduced by applying tf.serving during prediction time.

3. Checkpoints : Model Training Design Pattern

Resilience and fault tolerance are the major characteristics of a scalable system. A checkpoint is a snapshot of a model’s internal state (including weights, current learning rate), so that training can be resumed from this state at any later point in time. Power outages, OS faults, task preemptions, or any other unforeseen errors can happen during the training, especially with deep learning models that can result in losing entire time and resources. Checkpoints are enabled in majority of the deep learning frameworks including TensorFlow, Keras and PyTorch. One of the approach for implementing a checkpoint can be:

1. Keep a checkpoint at the end of every epoch

2. Save the model with the best accuracy

TensorFlow and Keras provide API’s such as tf.Estimator API and ModelCheckpoint API respectively and model weights can be saved in PyTorch to implement checkpoints.

4. Workflow Pipeline : Reproducibility Design Pattern

This design pattern aims to isolate and containerize the individual steps of a machine learning work flow into an organized workflow to ensure maintainability and scalability. The machine learning development workflow is generally monolithic, which contain a series of tasks from data collection to training a model and its evaluation. But the machine learning tasks are iterative in nature. But tracking all the small changes done in the workflow during the development process becomes complicated as the process is iterated more number of times. This introduces the concept of MLOps , which contain similar concepts as in DevOps such as continuous integration and continuous delivery (CI/CD) but also includes additional concepts of continuous evaluation and continuous training. The key difference in MLOps and DevOps is that, it is not only the code, but also the data which has to be continuously tested and validated in MLOps. Containerization of each of the tasks in the workflow is the methodology proposed to implement this design pattern. The authors propose a ML pipeline which is an executable DAG(Directed Acyclic Graph) of ML steps which are containers. The main problem it solves is the “Dependency Tracking”, such that only the dependent steps are re-run incase changes are made at a certain step of the pipeline ensuring reproducibility.

5. Explainable Predictions : Responsible AI Design Pattern

Machine learning models are black boxes in general. But having a clear understanding of the model behavior is very important to diagnose the errors and to identify potential biases to decide if they can be employed. Introducing explainability in machine learning is a major factor in Responsible AI. Hence, the key idea of this pattern is to interpret the machine learning models to understand why and how the model made the predictions in a certain way. But the current evaluation metrics only talk about the relative correctness of the predictions given the data set, but they don’t provide any insight on why the model arrived at that prediction. ELI5, SHAP and LIME are the most widely used Python libraries for model explainability.

Though the above techniques are implemented in most of the machine learning practices, defining design patterns helps to create general reusable solutions for the commonly occurring problems. They help in communication between engineers and quickly solve common problems by providing off-the-shelf solutions.

Happy Learning!!