Three ways to use custom validation metrics in tf.keras / TF2 | by Christian Freischlag | Towards Data Science

Keras offers a bunch of metrics to validate the test data set like accuracy, MSE or AUC. However, sometimes you need a custom metric to validate your model. In this post, I will show three different approaches to implement your metrics and use it within Keras.
While in the beginning there were only a few metrics included in Keras, today there are a lot of different ones available. For the Keras version bundled with TensorFlow 2 all the metrics can be found in tf.keras.metrics.
The Tensoflow Addons library makes some additional metrics available. Before starting to implement it on your own better check, if your metric is available there. To use tensorflow addons just install it via pip:
pip install tensorflow-addons
If you didn’t find your metrics there we can now look at the three options. Functions, Callbacks and Metrics objects.
The easiest way of defining metrics in Keras is to simply use a function callback. The function takes two arguments. The first parameter is the ground truth (y_true) and the second is the prediction from the model (y_pred). While running the validation, these arguments are Tensors, so we have to use the Keras Backend for calculation.
import tf.keras.backend as Kdef matthews_correlation(y_true, y_pred):    y_pred_pos = K.round(K.clip(y_pred, 0, 1))    y_pred_neg = 1 - y_pred_pos    y_pos = K.round(K.clip(y_true, 0, 1))    y_neg = 1 - y_pos    tp = K.sum(y_pos * y_pred_pos)    tn = K.sum(y_neg * y_pred_neg)    fp = K.sum(y_neg * y_pred_pos)    fn = K.sum(y_pos * y_pred_neg)    numerator = (tp * tn - fp * fn)    denominator = K.sqrt((tp + fp) * (tp + fn) * (tn + fp) * (tn + fn))    return numerator / (denominator + K.epsilon())
Using this function with Keras models:
model.compile(optimizer='rmsprop',              loss='binary_crossentropy',              metrics=[matthews_correlation])
And calling fit() now outputs:
245063/245063 [==============================] - 63s 256us/step - matthews_correlation: 0.0032 - val_matthews_correlation: 0.0039(Note that the name is the function name, while for validation data there is always the val_ prefix)
If you want to evaluate the model on a separate data set every N epochs you can use a custom callback. In this case, the AUC score from scikit-learn is used.
from sklearn.metrics import roc_auc_scorefrom tf.keras.callbacks import Callbackclass IntervalEvaluation(Callback):    def __init__(self, validation_data=(), interval=10):        super(Callback, self).__init__()        self.interval = interval        self.X_val, self.y_val = validation_data    def on_epoch_end(self, epoch, logs={}):        if epoch % self.interval == 0:            y_pred = self.model.predict_proba(self.X_val, verbose=0)            score = roc_auc_score(self.y_val, y_pred)            print("interval evaluation - epoch: {:d} - score: {:.6f}".format(epoch, score))
To train the model with this evaluation setup, the instance of the object is passed to Keras as a callback.
ival = IntervalEvaluation(validation_data=(x_validate, y_validate), interval=10)model.fit(x_train, y_train,          batch_size=8196,          epochs=25,          validation_data=[x_test, y_test],             class_weight=class_weight,          callbacks=[ival],          verbose=1 )
And the result looks like this:
interval evaluation - epoch: 0 - score: 0.545038interval evaluation - epoch: 10 - score: 0.724098interval evaluation - epoch: 20 - score: 0.731381
And finally, it’s possible to extend the metric object itself. (Credits to Geeocode from Stackoverflow)
class CategoricalTruePositives(tf.keras.metrics.Metric):    def __init__(self, num_classes, batch_size,                 name="categorical_true_positives", **kwargs):        super(CategoricalTruePositives, self).__init__(name=name, **kwargs)        self.batch_size = batch_size        self.num_classes = num_classes            self.cat_true_positives = self.add_weight(name="ctp", initializer="zeros")    def update_state(self, y_true, y_pred, sample_weight=None):             y_true = K.argmax(y_true, axis=-1)        y_pred = K.argmax(y_pred, axis=-1)        y_true = K.flatten(y_true)        true_poss = K.sum(K.cast((K.equal(y_true, y_pred)), dtype=tf.float32))        self.cat_true_positives.assign_add(true_poss)    def result(self):        return self.cat_true_positives
There is only one notable difference to the first example. The result function returns and the update_state method must be overridden (this is were the calculation happens). It also takes one additional argument (sample_weight). And finally the result tensor is returned by re result method.
model.compile(optimizer='adam',              loss='binary_crossentropy',              metrics=[CategoricalTruePositives(num_classes, batch_size)])
We have seen three different ways of implementing a custom validation metric. Hopefully, that helped to decide which way works for your use case and don’t forget to check, whether your metric is already available in the large amount of predefined ones in tf.keras.metrics.
This post was initially published on digital-thinking.de (Dec 19 2018)