Deep Learning With Weighted Cross Entropy Loss On Imbalanced Tabular Data Using FastAI | by Faiyaz Hasan | Towards Data Science

FastAI is an incredibly convenient and powerful machine learning library bringing Deep Learning (DL) to the masses. I was motivated to write this article while troubleshooting some issues related to training a model for a binary classification task. My objective is to walk you through the steps required to a build a simple and effective DL model using FastAI for a tabular, imbalanced dataset while avoiding the mistakes that I had made. The discussion in this article is organized according to the sections listed below.
DatasetSample CodeCode BreakdownComparison Of FastAI to PySpark ML & Scikit-LearnConclusion
Dataset
Sample Code
Code Breakdown
Comparison Of FastAI to PySpark ML & Scikit-Learn
Conclusion
The dataset comes from the context of ad conversions where the binary target variables 1 and 0 correspond to conversion success and failure. This proprietary dataset (no, I don’t own the rights) has some particularly interesting attributes due to its dimensions, class imbalance and rather weak relationship between the features and the target variable.
First, the dimensions of the data: this tabular dataset contains a fairly large number of records and categorical features that have a very high cardinality.
Note: In FastAI, categorical features are represented using embeddings which can improve classification performance on high cardinality features.
Second, the binary class labels are highly imbalanced since successful ad conversions are relatively rare. In this article we adapt to this constraint via an algorithm-level approach (weighted cross entropy loss functions) as opposed to a data-level approach (resampling).
Third, the relationship between the features and the target variable is rather weak. For example, a Logistic Regression model had a validation area under ROC curve of 0.74 after significant model tuning.
Dimensions: 17 features, 1 target variable, 3738937 rows
Binary target classes
Class imbalance ratio of 1:87
6 numerical features
8 categorical features
Categorical features have a combined cardinality of 44,000
This code was ran on a Jupyter Lab notebook on Google Cloud — AI Platform with the specs listed below.
4 N1-standard vCPUs, 15 GB RAM
1 NVIDIA Tesla P4 GPU
Environment: PyTorch 1.4
Operating System: Debian 9
The model training pipeline, which will be explained in the next section, is shown below.
Install both fastai and fastbook via the command terminal. For more details on the setup, checkout this link.
conda install -c fastai -c pytorch fastaigit clone https://github.com/fastai/fastbook.gitpip install -Uqq fastbook
Import the FastAI library and pandas into the notebook.
import pandas as pd from fastai.tabular.all import *
The column names had to be anonymized due to privacy reasons.
df = pd.read_csv('data/training.csv')# Categorical FeaturesCAT_NAMES = ['col_1', 'col_2', 'col_3', 'col_4',              'col_5', 'col_6', 'col_7', 'col_8'] # Continuous FeaturesCONT_NAMES = ['col_9', 'col_10', 'col_11',               'col_12', 'col_13', 'col_14'] # Target VariableTARGET = 'target'
Change the data type of the binary target variable to category.
df[TARGET] = df[TARGET].astype('category')
Pitfall #1: If the target variable data type is left as a numeric value, FastAI/PyTorch will treat it as such and yield a runtime error.
Next, list the data preprocessors, training/validation set split and create the tabular dataloader.
# Data Processorsprocs = [Categorify, FillMissing, Normalize] # Training/Validation Dataset 80:20 Split splits = RandomSplitter(valid_pct=0.2)(range_of(df)) dls = TabularDataLoaders.from_df(df,                                                                          y_names=TARGET,                                                                   cat_names=CAT_NAMES,                                                                  cont_names=CONT_NAMES,                                                                  procs=procs,                                                                  splits=splits)
Use dls.xs to see the transformed training data.
Construct Loss Function Weights
The class imbalances are used to create the weights for the cross entropy loss function ensuring that the majority class is down-weighted accordingly. The formula for the weights used here is the same as in scikit-learn and PySPark ML.
class_count_df = df.groupby(TARGET).count() n_0, n_1 = class_count_df.iloc[0, 0], class_count_df.iloc[1, 0] w_0 = (n_0 + n_1) / (2.0 * n_0)w_1 = (n_0 + n_1) / (2.0 * n_1) class_weights=torch.FloatTensor([w_0, w_1]).cuda()
Pitfall #2: Ensure that the class weights are converted to a float tensor and that cuda operations are enabled via .cuda(). Otherwise, you will get a type error.
TypeError: cannot assign 'list' object to buffer 'weight' (torch Tensor or None required)
Instantiate Area Under ROC Metric
roc_auc = RocAucBinary()
Pitfall #3: For binary class labels, use RocAucBinary() and NOT RocAuc() in order to avoid a value error.
ValueError: y should be a 1d array, got an array of shape (2000, 2) instead.
Instantiate Loss Function
loss_func = CrossEntropyLossFlat(weight=class_weights)
Pitfall #5: Use the FastAI cross entropy loss function as opposed to the PyTorch equivalent of torch.nn.CrossEntropyLoss() in order to avoid errors. The FastAI loss functions are listed here. Using the PyTorch cross entropy loss gave me the following runtime error.
RuntimeError: Expected object of scalar type Long but got scalar type Char for argument #2 'target' in call to _thnn_nll_loss_forward
Instantiate Learner
Use tabular_learner in FastAI to easily instantiate an architecture.
learn = tabular_learner(dls,                         layers=[500, 250],                            loss_func=loss_func,                         metrics=roc_auc)
Double check that learn is using the correct loss function:
learn.loss_func Out [1]: FlattenedLoss of CrossEntropyLoss()
Train the model over the desired number of epochs.
learn.fit_one_cycle(3)
The performance metric and the loss function values for the training and validation set is shown below.
That’s great! With minimal tuning, the FastAI model is performing better than the models built painstakingly using PySpark and Scikit-Learn.
In this section , we compare model performance and computation time of these three ML libraries.
Note: While the neural network performed well without extensive hyper-parameter tuning, PySpark ML and Scikit-Learn did not. As a result, I have added those times since they were relevant to training a descent model.
Model Training Time
FastAI — 6 minutesPySpark ML — 0.7 seconds + 38 minutes for hyper-parameter tuningScikit-Learn — 36 seconds + 8 minutes for hyper-parameter tuning (on a subsample of the data)
FastAI — 6 minutes
PySpark ML — 0.7 seconds + 38 minutes for hyper-parameter tuning
Scikit-Learn — 36 seconds + 8 minutes for hyper-parameter tuning (on a subsample of the data)
Area Under ROC Curve
FastAI — 0.75PySpark ML — 0.74Scikit-Learn — 0.73
FastAI — 0.75
PySpark ML — 0.74
Scikit-Learn — 0.73
For more details, check out my Github repo.
In this article, we saw the power of FastAI when it comes to quickly building DL models. Before playing around with FastAI, I would put off using neural networks until I had already tried Logistic Regression, Random Forests etc. because of the stigma that neural networks are difficult to tune and computationally expensive. However, with increased accessibility to GPUs via Google AI Platform Notebooks and the layered FastAI approach to deep learning, now it will surely be one of the first tools I reach for when it comes to classification tasks on large, complex datasets.