Question: I'm using Tensorflow/Keras 2.4.1 and I have a (unsupervised) custom metric that takes several of my model inputs as parameters such as: model = build_model() # returns a tf.keras.Model object my_metric = custom_metric(model.output, model.input[0], model.input[1]) model.add_metric(my_metric) [...] model.fit([...]) # training with fit  However, it happens that custom_metric is very expensive so I would like it to be computed during validation only. I found this answer but I hardly understand how I can adapt the solution to my metric that uses several model inputs as parameter since the update_state method doesn't seem flexible. In my context, is there a way to avoid computing my metric during training, aside from writing my own training loop ? Also, I am very surprised we cannot natively specify to Tensorflow that some metrics should only be computed at validation time, is there a reason for that ? In addition, since the model is trained to optimize the loss, and that the training dataset should not be used to evaluate a model, I don't even understand why, by default, Tensorflow computes metrics during training.

Answer: I think that the simplest solution to compute a metric only on the validation is using a custom callback. here we define our dummy callback: class MyCustomMetricCallback(tf.keras.callbacks.Callback): def __init__(self, train=None, validation=None): super(MyCustomMetricCallback, self).__init__() self.train = train self.validation = validation def on_epoch_end(self, epoch, logs={}): mse = tf.keras.losses.mean_squared_error if self.train: logs['my_metric_train'] = float('inf') X_train, y_train = self.train[0], self.train[1] y_pred = self.model.predict(X_train) score = mse(y_train, y_pred) logs['my_metric_train'] = np.round(score, 5) if self.validation: logs['my_metric_val'] = float('inf') X_valid, y_valid = self.validation[0], self.validation[1] y_pred = self.model.predict(X_valid) val_score = mse(y_pred, y_valid) logs['my_metric_val'] = np.round(val_score, 5)  Given this dummy model: def build_model(): inp1 = Input((5,)) inp2 = Input((5,)) out = Concatenate()([inp1, inp2]) out = Dense(1)(out) model = Model([inp1, inp2], out) model.compile(loss='mse', optimizer='adam') return model  and this data: X_train1 = np.random.uniform(0,1, (100,5)) X_train2 = np.random.uniform(0,1, (100,5)) y_train = np.random.uniform(0,1, (100,1)) X_val1 = np.random.uniform(0,1, (100,5)) X_val2 = np.random.uniform(0,1, (100,5)) y_val = np.random.uniform(0,1, (100,1))  you can use the custom callback to compute the metric both on train and validation: model = build_model() model.fit([X_train1, X_train2], y_train, epochs=10, callbacks=[MyCustomMetricCallback(train=([X_train1, X_train2],y_train), validation=([X_val1, X_val2],y_val))])  only on validation: model = build_model() model.fit([X_train1, X_train2], y_train, epochs=10, callbacks=[MyCustomMetricCallback(validation=([X_val1, X_val2],y_val))])  only on train: model = build_model() model.fit([X_train1, X_train2], y_train, epochs=10, callbacks=[MyCustomMetricCallback(train=([X_train1, X_train2],y_train))])  remember only that the callback evaluates the metrics one-shot on the data, like any metric/loss computed by default by keras on the validation_data. here is the running code.