Custom Loss and Custom Metrics Using Keras Sequential Model API
By: Vishnuvardhan Janapati
Tags: Deep Learning, TensorFlow, Inside Ai, Machine Learning Ai, Keras

3. How to define custom metrics for Keras models

Metrics and losses are recorded at the end of each epoch on the training and validation dataset (if provided). In this section, let’s review different ways of defining metric functions during model compilation. We will not go deep into discussing “what kind of metric function needs to be used for a given use case”, we will just focus on the implementation side of metric functions. The following figure shows four different ways to implement a metric in Keras.

Four different approaches of defining custom metrics

Why do we need custom metrics?

Currently, there are a good number of built-in metrics available under Keras to cover general use cases. However, if your use case is not a simple/general one then most probably you need to write a custom metric. Built-in metrics allow only target data (y_true), prediction data (y_pred), and class weights.

Approach #1: Built-in metrics (It is simple and not explained here)

Approach #2: Custom metric without external parameters

If you want to use built-in ops (such as K.mean, K.max, etc under Keras backend) to modify target/predicted data, then you can use approach #2 to estimate metrics.

Approach #3: A Custom metric with external parameters

If your use case requires you to use built-in ops and some other external object other than those mentioned above, then you can pass an external object and do some operations to modify y_true and/or y_pred before final computation of a metric. In this metric function, we need to define a wrapper function (that takes external parameters, in our case metric_with_params ) that wraps the loss function that can take only target ( y_true ) and predicted ( y_pred ) values as inputs. In Approach #3, we provided a constant ( magic_number ) as an external object and used it to update y_true.

Approach #4: Subclassing custom metric layer

If your use case is much more complex and couldn’t use any of the above three approaches, then go for defining a custom metric layer by subclassing the Metric class as shown below. This approach provides you more flexibility and control in defining the custom metric. In implementing the custom metrics layer, you need to define four methods as shown below. The short description of each of these four definitions is given below (from TensorFlow website)

__init__(self) , in which you will create state variables for your metric. update_state(self, y_true, y_pred, sample_weight=None) , which uses the targets y_true and the model predictions y_pred to update the state variables. result(self) , which uses the state variables to compute the final results. reset_states(self) , which reinitializes the state of the metric.

The idea behind the following simple example is to show the same metric value estimated using the above four approaches.

The output of the above code shows exactly the same metrics from the four approaches described above.

100/100 [==============================] — 0s 3ms/step — loss: 0.6877 — accuracy: 0.8044 — metric_no_params: 0.8044 — metric: 0.8044 — MyMetrics_layer: 0.8044

There are even more complex approaches to define custom metrics. However, they are not the scope of this article.