Handling data imbalance for a multi-label classification problem
By: Arjun Voruganti
Tags: Data Imbalance, Oversampling, Machine Learning, Classification, Multi Label

This post introduces my solution to multi-label classification data imbalance using data augmentation, namely oversampling with the multi-label SMOTE (MLSMOTE) algorithm.

Background

This is the first post of a series dedicated to training a machine learning model for my first Kaggle competition. As a current undergraduate student, my experience with machine learning is quite elementary. This series aims to highlight the steps I took that made my participation in the competition more effortless and beginner-friendly.

There are two main technologies/platforms that enabled me to build a well-performing model on my first try at a Kaggle competition:

trainML: A cloud GPU service providing access to multiple GPUs and plenty of cheap compute for notebook ML model training. trainML uses JupyterLab for their multi-GPU notebook environment, and boasts a superior pricing model as well as access to more powerful GPUs (GeForce RTX 2080 Ti as of writing this post) than competitors. hyperopt: An AutoML library for hyperparameter optimization and function minimization. It is especially useful for ML model training, where hyperopt’s fmin can identify the best hyperparameter values for model training given a range/search space of values to choose from.

The remainder of this series will demonstrate the implementation and use of these technologies in-depth. Ultimately, my goal is to demonstrate how trainML and hyperopt make ML model training more effortless while still outputting models that perform well in Kaggle competitions. This post introduces my particular competition dataset and my approach to mitigating class imbalance.

The competition

Source: Kaggle

I picked the Mechanisms of Action Kaggle competition, where the challenge is to train an ML model that assigns (single/multiple) labels to drugs based on their interactions with several compounds. Hence, this competition was a multi-label classification problem. This means that the target variable itself is a vector, with its length being the total number of labels.

There were two training datasets:

Training features: a list of drug samples and their interactions with several compounds (non-discrete feature values) Training targets: a list of those same drug samples and their (single/multiple) labels indicating their chemical makeup (0/1 binary target values)

Given a test feature dataset with new drug samples and their interactions, the task was to generate a targets dataset on those new samples. Each target value was to be a non-discrete value between 0 and 1, indicating the probability that a given sample has a positive response for each label. Kaggle then scored this dataset using a log-loss comparison to the true targets dataset.

Data imbalance

There were two levels of class imbalance evident upon inspection of the raw data.

Firstly, there was a significant number of samples that had target vectors of all 0's, i.e. without any positive classification labels.

Secondly, of those samples that had one (or more) positive labels, there were a handful of labels that were largely overrepresented compared to the other labels (majority/minority labels). There were 50 minority labels represented by 20 or less drug samples (<0.1% of the dataset).

When labels are unevenly distributed, deep learning algorithms tend to turn a blind eye to those minority labels that often still matter for making predictions on test data. The solution is to augment and adjust the distribution of the original training dataset.

Data augmentation using MLSMOTE algorithm

For this competition, I chose to oversample the training dataset. In data oversampling, a random set of “synthetic”data points gets added to the original dataset. These new data points have the minority labels, thereby making the overall label distribution more equal.

Originally, I had hoped to utilize the Python package imbalanced-learn for its easy to use, built-in data augmentation algorithms. However, their data oversampling algorithm offerings were geared towards binary classification problems and not to multi-label classification problems.

My solution was to hard-code the multi-label SMOTE (MLSMOTE) algorithm, an extension of the SMOTE algorithm that is popularly used for oversampling in binary classification. MLSMOTE takes three main steps:

Identify the minority labels and select data points that have those labels. Use those selected data points as reference points. Draw synthetic data points arbitrarily near the reference points in feature space. Classify the synthetic data points according to the minority labels of their respective nearby reference points.

A demonstration of the SMOTE algorithm. For MLSMOTE, we’ll have to imagine that the data points can be circles, squares, circles+squares, and other combinations of shape “labels.” Source

My final product was an adaptation of this implementation of MLSMOTE, which made its way into my Kaggle competition notebook after data preprocessing.

Opportunities for automation

My challenge was determining the number of synthetic data points to oversample by in order to most effectively augment the data. In other words, the performance of my trained model was somewhat a function of the number of data points I chose to oversample by, among other things. With limited ML experience, I was unsure of what would be the most effective amount to oversample by, or whether I was oversampling by too much.

My next article addresses how I didn’t have to guess the best parameters for running the MLSMOTE algorithm, and how I rather got away with using the function minimizer hyperopt, as well as the help of trainML to provide me sufficient GPU power to run hyperopt externally from Kaggle.