Making Your Loss Function Count. Some errors are more costly than... | by Kieran | Towards Data Science

George Orwell’s novella Animal Farm includes the memorable line...
all animals are equal, but some animals are more equal than others 1
Orwell may have been referring to hypocrisy, power, and privilege in society, but if you replace the word animals with errors, it starts to become very relevant to machine learning.
Now that I’ve finished pretending to be well-read, let’s get more specific.
Explaining the concept of false positives and negatives is a popular interview question because they are so important when applying a classification algorithm in practice.
I still sometimes find myself hastily consulting Wikipedia just before a meeting to triple-check that I’m using terms like precision and recall the right way round. For this article, you should only need to remember...
False positives are when the model predicts that a condition exists, but it does not, in reality (e.g. a fire alarm going off when there is no fire).
False negatives are when the model predicts that the condition is not present, but in actual fact, it is (e.g. a smoke alarm fails to go off when there is a fire).
With the definitions out of the way, let’s imagine that you work for a bank and your boss has asked you to build a model to identify fraudulent customers. Before any modelling takes place, it would be important to ask...
What happens if my model gets it wrong?
I don’t mean asking this from a statistical point of view, but instead, really understanding what it means for the business.
The answer you receive will determine how you evaluate and deploy your model.
In Scenario 1, your model does not need to be as confident when marking a customer as fraudulent given that a more detailed human review will follow, and you would expect false positives to be filtered out by the fraud team.
Conversely, in Scenario 2, it is a riskier outcome when a customer is flagged because they are automatically suspended. If too many false positives slip through, then you are suspending innocent customers, giving them a nasty experience, and generating bad press.
Identifying the sweet spot for how strict or relaxed your model should be is quite a balancing act because targeting a reduction in false positives inherently leads to an increase in false negatives and vice versa.
Given the trade-off between false positive and negative rates, a model’s output can be used more creatively.
Binary classification models often classify anything that has over 50% probability, but if you are concerned with the false positive rate being too high, you could use a higher threshold e.g. only classify fraudulent customers if their predicted probability exceeds 80%.
Or the business could follow a hybrid approach between Scenario 1 and 2, where customers with a predicted probability of 80% or above are automatically suspended, and anyone between 50% and 80% is shared with the fraud team for more rigorous review.
This kind of strategy often works well enough, but ultimately they involve training a machine learning algorithm that assumes all types of error are equally important and then massaging its output to correct that assumption.
Fortunately, it is possible to directly influence how a variety of machine learning algorithms learn, and place more importance on a specific error type. Some of the techniques for handling class imbalance can also serve a similar purpose of targeting false positives or negatives, but this article focuses on achieving this goal via the loss function.
When it comes to deep learning, the loss function determines what the algorithm is trying to minimise as it learns and iteratively improves. Loss functions often default to mean squared error (regression) and cross-entropy loss (classification), but there is plenty of funkier options which can work better in certain use cases.
Regardless of the function itself, however, they all share a purpose in that they provide a feedback mechanism for the model to check how well it is performing, find the gradient of the loss function, and use that gradient to update the network’s weights to reduce loss as quickly as possible.
Even though we’re now talking about deep learning and much more complex and powerful models, they would still fail to distinguish between the type of error they are committing.
Fortunately, TensorFlow allows you to take control of this.
Just before we get to the code, let’s take a new scenario (I promise this is the last one).
You are working for a leisure centre where customers pay a monthly subscription, but they have the option of changing which price tier they subscribe to at the end of each month.
As with most businesses, it’s helpful to know whether customers are spending more or less over time, and forecast what they are likely to do in the future.
To offer that capability, you are asked to build a model which can learn from a customer’s characteristics and predict whether they will trade up, down, or stay on the same price plan.
By taking a snapshot of past behaviour i.e. whether a customer was using the gym more or less vs the last month and how long they had spent on the same plan, a model could be trained to predict one of three classes for what the customer will do next month:
Spend more and trade Up
Stay on the same plan and remain Stable
Spend less and trade Down
Naturally, you would like the model to be as accurate as possible across the board. Yet the model’s output might be used by the Finance Team to determine how much revenue will be coming in and how much they can afford to reinvest.
In this context, it is far more damaging to overestimate how many customers will trade-up, as the business will be expecting more revenue than it achieves. Whilst it isn’t great to overestimate customers trading down, at least it leaves the Finance Team in a nicer position of having more money coming in than expected.
Armed with this commercial context, you can directly manipulate how the model trains to reflect the cautiousness the Finance Team desires.
We shall use TensorFlow and the Keras API to demonstrate how this can be achieved. If you wish to follow along and execute the code yourself, you can follow the instructions here.
Have a read of the code below to get a taste of Tensorflow’s flexibility.
This notebook accompanies the medium article Making Your Loss Function Count.
The code within highlights how to use a custom loss function in Tensorflow in a hypothetical scenario, in which you  work for a leisure centre where customers can sign up for different price tiers each month, and your job is to predict whether they will trade up, trade down, or remain at the same price tier in the next month.
import matplotlib.pyplot as plt
import numpy as np
import os
import pandas as pd
import seaborn as sns
from sklearn import metrics, model_selection, preprocessing
import tensorflow as tf
from tensorflow.keras import losses
from tensorflow.keras import utils as keras_utils
from tensorflow.keras.layers import Input, Dense, Dropout
from tensorflow.keras.models import Model, Sequential
from typing import Tuple

number_of_observations_per_class = 10000

dict_trade_down = {
    'Shift_In_Visits_Vs_Last_Month': np.random.uniform(low=0.7, high=1.1, size=number_of_observations_per_class),
    'Months_On_Price_Tier': np.random.randint(low=0, high=3, size=number_of_observations_per_class),
    'Class': np.repeat(a='Down', repeats=number_of_observations_per_class)
}

df_trade_down = pd.DataFrame(data=dict_trade_down)

dict_stable = {
    'Shift_In_Visits_Vs_Last_Month': np.random.uniform(low=0.7, high=1.15, size=number_of_observations_per_class),
    'Months_On_Price_Tier': np.random.randint(low=0, high=5, size=number_of_observations_per_class),
    'Class': np.repeat(a='Stable', repeats=number_of_observations_per_class)
}

df_stable = pd.DataFrame(data=dict_stable)

dict_trade_up = {
    'Shift_In_Visits_Vs_Last_Month': np.random.uniform(low=0.7, high=1.2, size=number_of_observations_per_class),
    'Months_On_Price_Tier': np.random.randint(low=0, high=3, size=number_of_observations_per_class),
    'Class': np.repeat(a='Up', repeats=number_of_observations_per_class)
}

df_trade_up = pd.DataFrame(data=dict_trade_up)

df_all = pd.concat([df_trade_down, df_stable, df_trade_up]).sample(frac=1)

df_all.head()

label_encoder = preprocessing.LabelEncoder()
label_encoder = label_encoder.fit(df_all['Class'])

print('Label mapping: ')
label_mapping = dict(zip(label_encoder.classes_, range(len(label_encoder.classes_))))
print(label_mapping)

# Transform the original datasets
df_all['Class_Integer'] = label_encoder.transform(df_all['Class'])

df_all[['Class', 'Class_Integer']].head(10)

Label mapping: 
{'Down': 0, 'Stable': 1, 'Up': 2}

No test dataset is taken in this demo for simplicity
features = df_all.loc[:, ['Shift_In_Visits_Vs_Last_Month', 'Months_On_Price_Tier']]
labels = df_all.loc[:, 'Class_Integer']

X_train, X_validation, y_train, y_validation = model_selection.train_test_split(features, labels, test_size=0.2)

y_train_encoded = keras_utils.to_categorical(y_train, num_classes=3)
y_validation_encoded = keras_utils.to_categorical(y_validation, num_classes=3)

scenario_risk = {}

scenario_risk['prediction_matches_reality'] = 1.0

scenario_risk['less_spend_than_expected'] = 3.0

scenario_risk['much_less_spend_than_expected'] = 9.0

scenario_risk['more_spend_than_expected'] = 1.25

scenario_risk['much_more_spend_than_expected'] = 1.05

print('Scenario risks: \n')
for key, value in scenario_risk.items(): print(f'{key}: {value}')

Scenario risks: 

prediction_matches_reality: 1.0
less_spend_than_expected: 3.0
much_less_spend_than_expected: 9.0
more_spend_than_expected: 1.25
much_more_spend_than_expected: 1.05

We will want to apply these weights to help with back-propagation and forcing the model to place more importance under certain scenarios. We want to avoid having weights greater than 1 as it can distort the learning process and apply weight changes which are too aggressive. As such, divide each weight by the maximum weight to have a normalised version between 0 and 1.
scenario_risk_normalised = {}
worst_case_scenario_risk = np.array(list(scenario_risk.values())).max()

for key, value in scenario_risk.items():
    scenario_risk_normalised[key] = value / worst_case_scenario_risk
    
print('New scenario weights: \n')
for key, value in scenario_risk_normalised.items(): print(f'{key}: {value}')

New scenario weights: 

prediction_matches_reality: 0.1111111111111111
less_spend_than_expected: 0.3333333333333333
much_less_spend_than_expected: 1.0
more_spend_than_expected: 0.1388888888888889
much_more_spend_than_expected: 0.11666666666666667

def _convert_scenario_risk_to_tensor(scenario: str) -> tf.Tensor:
    """
    Look up how risky the scenario is when comparing the model prediction to reality, and return a weight score
    indicating how much emphasis the model should place on learning from that particular observation.

    Parameters
    ----------
    scenario : str
        Key of dictionary which shows the risk attached to each scenario. Should be one of...
            'prediction_matches_reality'
            'less_spend_than_expected'
            'much_less_spend_than_expected'
            'more_spend_than_expected'
            'much_more_spend_than_expected'

    Returns
    -------
    tf.Tensor
        Weight which signifies how bad the scenario is for the business, and thus how to update the loss which the
        model is trying to improve.
    """

    return tf.cast(x=tf.constant(scenario_risk_normalised[scenario]), dtype=tf.float32)

def _get_loss_adjustment_for_scenario(actual_vs_predicted_class: Tuple[tf.Tensor, tf.Tensor]) -> tf.Tensor:
    """
    Return a weight according to how damaging the scenario is to the business.

    Parameters
    ----------
    actual_vs_predicted_class : Tuple[tf.Tensor, tf.Tensor]
       The actual class, actual_vs_predicted_class[0], and the predicted class, actual_vs_predicted_class[1], for an
       observation.

    Returns
    -------
    float
        Weight which signifies how bad the scenario is for the business, and thus how to upweight the loss
        which the model should be trying to improve.
    """

    actual_class = actual_vs_predicted_class[0]
    predicted_class = actual_vs_predicted_class[1]

    # Retrieve the appropriate weighting based on how the prediction compares with reality
    # The labels are:
    # 0: Trade Down, 1: Stable, 2: Trade Up
    return tf.case(
        [
            (tf.equal(actual_class, predicted_class),
             lambda: _convert_scenario_risk_to_tensor('prediction_matches_reality')),

            (tf.logical_and(tf.equal(actual_class, 0), tf.equal(predicted_class, 1)),
             lambda: _convert_scenario_risk_to_tensor('less_spend_than_expected')),

            (tf.logical_and(tf.equal(actual_class, 1), tf.equal(predicted_class, 2)),
             lambda: _convert_scenario_risk_to_tensor('less_spend_than_expected')),

            (tf.logical_and(tf.equal(actual_class, 0), tf.equal(predicted_class, 2)),
             lambda: _convert_scenario_risk_to_tensor('much_less_spend_than_expected')),

            (tf.logical_and(tf.equal(actual_class, 1), tf.equal(predicted_class, 0)),
             lambda: _convert_scenario_risk_to_tensor('more_spend_than_expected')),

            (tf.logical_and(tf.equal(actual_class, 2), tf.equal(predicted_class, 1)),
             lambda: _convert_scenario_risk_to_tensor('more_spend_than_expected')),

            (tf.logical_and(tf.equal(actual_class, 2), tf.equal(predicted_class, 0)),
             lambda: _convert_scenario_risk_to_tensor('much_more_spend_than_expected')),
        ]
    )

def custom_cross_entropy(y_actual: tf.Tensor, y_prediction: tf.Tensor) -> tf.Tensor:
    """
    Calculate cross entropy loss, but weighted according to how risky the scenario is.

    Parameters
    ----------
    y_actual : tf.Tensor
        One-hot encoded representation of the true class for each observation.
    y_prediction : tf.Tensor
        The predicted probability that the observation belongs to each class.
        
    Returns
    -------
    tf.Tensor
        Weighted version of cross-entropy loss depending on how the prediction compares to reality.
    """

    standard_cross_entropy = losses.categorical_crossentropy(y_true=y_actual, y_pred=y_prediction)

    # The label is one hot encoded, so identify the true class by which element contains the maximum value (1)
    actual_class = tf.math.argmax(input=y_actual, axis=1)
    actual_class = tf.cast(x=actual_class, dtype=tf.float32)

    # The prediction is a distribution of probabilities per class,
    # so identify the predicted class by which element contains the maximum probability
    predicted_class = tf.math.argmax(input=y_prediction, axis=1)
    predicted_class = tf.cast(x=predicted_class, dtype=tf.float32)

    actual_vs_predicted_class = tf.stack(values=[actual_class, predicted_class], axis=1)

    # Calculate the weighting that should be applied to each observation in the data
    weighting = tf.map_fn(fn=_get_loss_adjustment_for_scenario, elems=actual_vs_predicted_class)

    # Updated the cross entropy loss based on the weighting
    return tf.math.multiply(standard_cross_entropy, weighting)

# Suppress tensorflow warnings
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'

num_features = X_train.shape[1]

loss = 'categorical_crossentropy'

standard_model = Sequential()

standard_model.add(Input(shape=(num_features, )))
standard_model.add(Dense(64, activation='relu'))
standard_model.add(Dense(32, activation='relu'))
standard_model.add(Dense(len(label_encoder.classes_), activation='softmax'))

standard_model.summary()

standard_model.compile(
    optimizer='adam', 
    loss=loss,
    metrics=['categorical_accuracy', 'categorical_crossentropy']
)

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense (Dense)                (None, 64)                192       
_________________________________________________________________
dense_1 (Dense)              (None, 32)                2080      
_________________________________________________________________
dense_2 (Dense)              (None, 3)                 99        
=================================================================
Total params: 2,371
Trainable params: 2,371
Non-trainable params: 0
_________________________________________________________________

loss = custom_cross_entropy

custom_model = Sequential()

custom_model.add(Input(shape=(num_features, )))
custom_model.add(Dense(64, activation='relu'))
custom_model.add(Dense(32, activation='relu'))
custom_model.add(Dense(len(label_encoder.classes_), activation='softmax'))

custom_model.summary()

custom_model.compile(
    optimizer='adam', 
    loss=loss,
    metrics=['categorical_accuracy', 'categorical_crossentropy']
)

Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_3 (Dense)              (None, 64)                192       
_________________________________________________________________
dense_4 (Dense)              (None, 32)                2080      
_________________________________________________________________
dense_5 (Dense)              (None, 3)                 99        
=================================================================
Total params: 2,371
Trainable params: 2,371
Non-trainable params: 0
_________________________________________________________________

epochs = 20
verbosity = 0

standard_model.fit(
    x=X_train, 
    y=y_train_encoded, 
    epochs=epochs, 
    verbose=verbosity,
    validation_data=(X_validation, y_validation_encoded),
    batch_size=2048
)

<tensorflow.python.keras.callbacks.History at 0x141295d00>
custom_model.fit(
    x=X_train, 
    y=y_train_encoded, 
    epochs=epochs, 
    verbose=verbosity,
    validation_data=(X_validation, y_validation_encoded),
    batch_size=128
)

<tensorflow.python.keras.callbacks.History at 0x141a0e5e0>
validation_standard_predictions = standard_model.predict(X_validation, batch_size=2048)
validation_custom_predictions = custom_model.predict(X_validation, batch_size=2048)

validation_standard_predictions_class = np.argmax(validation_standard_predictions, axis=1)
validation_custom_predictions_class = np.argmax(validation_custom_predictions, axis=1)

validation_standard_predictions_class = label_encoder.inverse_transform(validation_standard_predictions_class)
validation_custom_predictions_class = label_encoder.inverse_transform(validation_custom_predictions_class)

validation_true_class = label_encoder.inverse_transform(y_validation)

standard_predicted_class_count = np.unique(ar=validation_standard_predictions_class, return_counts=True)

standard_predicted_class_count = pd.DataFrame({
    'Model': ['Standard'] * 3, 
    'Class': standard_predicted_class_count[0], 
    'Count': standard_predicted_class_count[1]
})


custom_predicted_class_count = np.unique(ar=validation_custom_predictions_class, return_counts=True)

custom_predicted_class_count = pd.DataFrame({
    'Model': ['Custom'] * 3, 
    'Class': custom_predicted_class_count[0], 
    'Count': custom_predicted_class_count[1]
})

predicted_class_count = pd.concat([standard_predicted_class_count, custom_predicted_class_count])

predicted_class_count.head(6)

plt.rcParams['figure.figsize'] = [20, 10]
plt.rcParams['font.size'] = 20

ax = sns.barplot(
    x='Class',
    y='Count',
    hue='Model',
    data=predicted_class_count
)

ax.set_title(label='Predicted Class by Model Type', pad=20, fontsize=30)
ax.set_xlabel(xlabel='Predicted Class', labelpad=20, fontsize=24)
ax.set_ylabel(ylabel='Number of Observations', labelpad=20, fontsize=24);

 

Hopefully, this has helped demonstrate the situations in which you may want a custom loss function, and how you go about implementing it.
But more importantly, never forget the key question...
What happens if my model gets it wrong?
Keep this top of mind and you will transform yourself from a Data Scientist who works well in theory to one who works well in practice.
Feel free to check out my other articles:
A simple trick to avoid running out of memory
How to Find the Right Number of Clusters
How to Manage a Junior Data Scientist
The Power of Permutation Testing
[1] G. Orwell. Animal farm: A fairy story (2018). London: Penguin Books.