Distribution-based loss functions for deep learning models | by Andrea C. | Towards Data Science

Information is made of data. During training step, an artificial neural network learns to map (predict) a set of inputs to a set of outputs from a labeled dataset. Computing the optimal weights is an optimization problem and it is usually solved by the stochastic gradient descent: weights are updated using the backpropagation of prediction error. The gradient descent algorithm updates weights navigating down the gradient (or slope) of the error, so that it can reduce the error of the next prediction. This is, in their very essence, how neural networks work.
When looking for the optimal weights, learning algorithms need a special object to evaluate the fitness of a candidate set of weights: this is the objective function.
Depending on the context, an objective function can be maximized or minimized. When dealing with deep learning models, experts prefer to reason in terms of error, so their goal is to minimize the objective function. Thus, objective function is called loss function and its values (i.e. the errors) are simply called losses. Loss functions are critical to ensure an adequate mathematical representation of the model response and their choice must be carefully considered as it must properly fit the model domain and its classification goals.
Definition and application of loss functions has started with standard machine learning methods. At the time, these functions were based on the distribution of labels, and it is for this reason that following functions are said to be distribution-based loss functions.
Our discussion begins precisely from here and, specifically, with the concept of cross-entropy.
Note: all code used for the plots and for the case study is available on my personal github: https://github.com/andrea-ci/misc-stuff/tree/master/nn-metrics.
According to the information theory, given a discrete random variable x, the entropy H(x) (also referred to as Shannon entropy, by its creator Claude Shannon) is defined as the expected value of logarithm of the inverse of probability:
Intuitively, entropy measures the uncertainty related to the possible outcomes of a random process: the more “surprising” they are for the observer, the higher the entropy is.
Note that the logarithm is in base 2 because in the context of information theory we are interested to the number of bits needed to encode the information carried by the random process.
Now suppose that we have two probability distributions, p and q, defined on the same random variable x. We want to measure the average number of bits needed to identify an event drawn from the sample space of x if the coding scheme is optimized for q rather than for p (that would be the true distribution).
In case the possible outcomes are only two, as in the previous example with the Bernoulli variable, we have the following definition for the binary cross-entropy loss function:
Having a binary scenario permits to simplify the equation so that we have only one argument, pt, which represents the value of probability assigned by the model to the true class (i.e. class to which the sample actually belongs).
Binary cross-entropy is widely used as loss function as it works well for many classification tasks. As a matter of fact, it is a fundamental baseline for distribution-based loss functions.
In the figure, yt is the class label of a sample in a binary classification task, and yp is the probability assigned by the model to that class. The function heavily penalizes the errors of prediction in which a wrong decision is made with high confidence (i.e. yp approaching 0), while it becomes zero when the prediction approaches true value of 1 (i.e. a correct classification is made).
A note: for sake of simplicity, we are going to keep both discussion and examples related to a binary classification task. However, the considerations made so far and in the following are valid, and naturally apply to, the case of multi-label classification as well.
In practice we must often deal with strongly unbalanced datasets: if, for example, we want to train a model to recognize ships in the middle of the ocean, the number of positive pixels, that is those that belong to the class “ship”, will be a very small percentage with respect to all the pixels.
It would be therefore useful to have loss functions able to handle different class labels in a different way, based on their abundance in the dataset or, in other words, according to their a priori classification probabilities.
Two variants of binary cross-entropy have been developed for this purpose: the Weighted Binary Cross-Entropy and the Balanced Cross-Entropy.
Weighted binary cross-entropy (WBCE) uses a coefficient to weight positive samples and it is usually preferred when data exhibit a significant skewness:
Balanced binary cross-entropy (BBCE) is similar to weighted cross-entropy, but in this case also negative samples are affected by a weight coefficient as in the following:
Finally, also Focal Loss (FL) can be seen as variation of binary cross-entropy. This function works well with highly imbalanced datasets because it weights the contribution of samples correctly classified and allows to better learn the hard samples (i.e. positive samples that are not detected):
Here λ is an hyper-parameter that we can use to calibrate importance of misclassified samples. Intuitively, λ decrease the penalty of easy samples by extending the range in which they receive low values of loss.
Note that when λ=0 we get the standard cross-entropy loss.
In addition, focal loss can take also another parameter, commonly referred to as α parameter, that allows to further balance the error term:
Let’s make an example where the choice of focal loss allows us to solve a problem of binary classification on an extremely unbalanced dataset.
To this purpose we consider the Credit Card Fraud Detection dataset hosted on Kaggle. As stated on their page:
“The dataset contains transactions made by credit cards in September 2013 by European cardholders.This dataset presents transactions that occurred in two days, where we have 492 frauds out of 284,807 transactions. The dataset is highly unbalanced, the positive class (frauds) account for 0.172% of all transactions.”
It goes without saying that the goal here is to detect the (very) small subset of fraudulent operations among all total transactions. In the following we give a brief overview of the followed steps.
First of all, we load the dataset and perform some cleanup in order to prepare data for model training.
## DATA LOADING/CLEANSING## Load data from CSV file.df_raw = pd.read_csv(‘creditcardfraud/creditcard.csv’)n_samples = len(df_raw)print(f’Num. of samples: {n_samples}.’)# Check size of samples.df_pos = df_raw[df_raw[‘Class’] == 1]n_pos_samples = len(df_pos)pos_ratio = 100 * n_pos_samples / n_samplesprint(f’Num. of positive samples: {n_pos_samples} ({pos_ratio:.2f}% of total).’)# Drop useless data and convert amount to log space.df_cleaned = df_raw.copy()df_cleaned.pop(‘Time’)df_cleaned[‘log-amount’] = np.log(df_cleaned.pop(‘Amount’) + 0.001)# Double train/test split for testing and validation data.df_train, df_test = train_test_split(df_cleaned, test_size = 0.2, shuffle = True)df_train, df_valid = train_test_split(df_train, test_size = 0.2, shuffle = True)print(f’Size of training data: {len(df_train)}.’)print(f’Size of validation data: {len(df_valid)}.’)print(f’Size of test data: {len(df_test)}.’)# Extract labels and features from data.labels_train = np.array(df_train.pop(‘Class’))labels_valid = np.array(df_valid.pop(‘Class’))labels_test = np.array(df_test.pop(‘Class’))features_train = np.array(df_train)features_valid = np.array(df_valid)features_test = np.array(df_test)# Normalize data.scaler = StandardScaler()features_train = scaler.fit_transform(features_train)features_valid = scaler.transform(features_valid)features_test = scaler.transform(features_test)# Enforce lower/upper bounds.features_train = np.clip(features_train, -5, 5)features_valid = np.clip(features_valid, -5, 5)features_test = np.clip(features_test, -5, 5)n_features = features_train.shape[-1]
We can see that dataset is actually extremely unbalanced, with positive samples accounting for only 0.17% of the total as anticipated.
With Keras, we setup a simple model and we train it using binary cross-entropy as loss function. This is our baseline model. Then we adopt focal loss function instead and we compare the performances obtained.
As evaluation metrics we consider true positives, false negatives, and recall metrics, because we are interested to measure how well the model predicts frauds.
In fact using accuracy, in this case, would not be a proper choice: a “model” that always predicts false (that is, no fraud) would get more than 99.8% accuracy on this dataset. This is why it’s hard to train a model on such unbalanced data.
## MODEL TRAINING## Model parameters.opt = Adam(learning_rate = 1e-3)metrics = [    TruePositives(name = 'tp'),    FalseNegatives(name = 'fn'),    Recall(name = 'recall')]losses = [    BinaryCrossentropy(),    SigmoidFocalCrossEntropy(gamma = 2, alpha = 4)]loss_names = [    'binary cross-entropy',    'focal loss']logs_loss = []logs_recall = []for loss in losses:    # Setup/compile the model.    model = Sequential()    model.add(Dense(16, input_dim = n_features, activation = 'relu',        kernel_initializer = 'he_uniform'))    model.add(Dropout(0.5))    model.add(Dense(1, activation = 'sigmoid'))    model.compile(optimizer = opt, loss = loss, metrics = metrics)    # Fit the model.    logs = model.fit(features_train, labels_train, validation_data = (features_valid,        labels_valid), epochs = EPOCHS, verbose = 0)    logs_loss.append(logs.history['loss'])    logs_recall.append(logs.history['recall'])    # Evaluate the model.    eval_train = model.evaluate(features_train, labels_train, verbose = 0)    eval_test = model.evaluate(features_valid, labels_valid, verbose = 0)table = PrettyTable()    table.field_names = ['Data', 'Loss', 'TruePositives', 'FalseNegatives', 'Recall']    for stage, eval_info in zip(('training', 'test'), (eval_train, eval_test)):        row = [stage]        for ii, lbl in enumerate(model.metrics_names):            row.append(f'{eval_info[ii]:.3f}')        table.add_row(row)    print('\n')    print(table)
Having trained and validated both configurations, we can compare the results obtained. From the baseline model we get:
Using focal loss instead, classification performs significantly better and all fraud cases are correctly detected:
Let’s also compare the behavior of the two loss functions.
As expected, values of focal loss are lower than those of cross-entropy. Focal loss down-weights the loss of positive samples (frauds) that are misclassified, thus “encouraging” the model to increase sensitivity to fraud cases.
[1] Tsung-Yi Lin, Priya Goyal et al., Focal Loss for Dense Object Detection
[2] Hichame Yessou et al., A Comparative Study of Deep Learning Loss Functions for Multi-Label Remote Sensing Image Classification
[3] Multi-class classification with focal loss for imbalanced datasets
[4] Classification on imbalanced data
[5] Kaggle dataset on Credit Card Fraud Detection