A Practical Introduction to Keras Callbacks in TensorFlow 2 | by B. Chen | Towards Data Science

When training a machine learning model, we would like to have the ability to monitor the model performance and perform certain actions depending on those performance measures. That’s where Keras Callbacks come in.
Callbacks are an important type of object TensorFlow and Keras that are designed to be able to monitor the performance in metrics at certain points in the training run and perform some action that might depend on those performance in metric values.
In this article, we’ll explore the following popular Keras Callbacks APIs with the help of some examples.
EarlyStopping: a callback designed for early stopping.CSVLogger: a callback streams epoch results to a CSV file.ModelCheckpoint : a callback to save the Keras model or model weight during trainingReduceLROnPlateau : a callback to reduce the learning rate when a metric has stopped improving.LearningRateScheduler: a callback for learning rate schedules.LambdaCallback: a callback for creating custom callbacks on-the-fly.
EarlyStopping: a callback designed for early stopping.
CSVLogger: a callback streams epoch results to a CSV file.
ModelCheckpoint : a callback to save the Keras model or model weight during training
ReduceLROnPlateau : a callback to reduce the learning rate when a metric has stopped improving.
LearningRateScheduler: a callback for learning rate schedules.
LambdaCallback: a callback for creating custom callbacks on-the-fly.
Please check out my Github repo for source code.
EarlyStopping is a built-in callback designed for early stopping. First, let’s import it and create an early stopping object:
from tensorflow.keras.callbacks import EarlyStoppingearly_stopping = EarlyStopping()
EarlyStopping() has a few options and by default:
monitor='val_loss': to use validation loss as performance measure to terminate the training.
patience=0: is the number of epochs with no improvement. The value 0 means the training is terminated as soon as the performance measure gets worse from one epoch to the next.
Next, we just need to pass the callback object to model.fit() method.
history = model.fit(    X_train,     y_train,     epochs=50,     validation_split=0.20,     batch_size=64,     verbose=2,    callbacks=[early_stopping])
You can see that early_stopping get passed in a list to the callbacks argument. It is a list because in practice we might be passing a number of callbacks for performing different tasks, for example, debugging and learning rate schedules.
By executing the statement, you should get an output like below:
Train on 8000 samples, validate on 2000 samplesEpoch 1/508000/8000 - 6s - loss: 1.5632 - accuracy: 0.5504 - val_loss: 1.1315 - val_accuracy: 0.6605............Epoch 10/508000/8000 - 2s - loss: 0.5283 - accuracy: 0.8213 - val_loss: 0.5539 - val_accuracy: 0.8170Epoch 11/508000/8000 - 2s - loss: 0.5141 - accuracy: 0.8281 - val_loss: 0.5644 - val_accuracy: 0.7990
The training gets terminated at Epoch 11 due to the increase of val_loss value and that is exactly the conditions monitor='val_loss' and patience=0.
It’s often more convenient to look at a plot, let’s run plot_metric(history, 'loss') to get a clear picture. In the below graph, validation loss is shown in orange and it’s clear that validation error increases at Epoch 11.
Apart from the options monitor and patience we mentioned early, the other 2 options min_delta and mode are likely to be used quite often.
EarlyStopping(    monitor='val_loss',     patience=0,     min_delta=0,     mode='auto')
monitor='val_loss': to use validation loss as performance measure to terminate the training.
patience=0: is the number of epochs with no improvement. The value 0 means the training is terminated as soon as the performance measure gets worse from one epoch to the next.
min_delta: Minimum change in the monitored quantity to qualify as an improvement, i.e. an absolute change of less than min_delta, will count as no improvement.
mode='auto': Should be one of auto, min or max. In 'min' mode, training will stop when the quantity monitored has stopped decreasing; in 'max' mode it will stop when the quantity monitored has stopped increasing; in 'auto' mode, the direction is automatically inferred from the name of the monitored quantity.
And here is an example of a customized early stopping:
custom_early_stopping = EarlyStopping(    monitor='val_accuracy',     patience=3,     min_delta=0.001,     mode='max')
monitor='val_accuracy' to use validation accuracy as performance measure to terminate the training. patience=3 means the training is terminated as soon as 3 epochs with no improvement. min_delta=0.001 means the validation accuracy has to improve by at least 0.001 for it to count as an improvement. mode='max' means it will stop when the quantity monitored has stopped increasing.
Let’s go ahead, run it with the customized early stopping, and plot the accuracy.
Train on 8000 samples, validate on 2000 samplesEpoch 1/50......Epoch 12/508000/8000 - 2s - loss: 0.5043 - accuracy: 0.8290 - val_loss: 0.5311 - val_accuracy: 0.8250Epoch 13/508000/8000 - 3s - loss: 0.4936 - accuracy: 0.8332 - val_loss: 0.5310 - val_accuracy: 0.8155Epoch 14/508000/8000 - 2s - loss: 0.4835 - accuracy: 0.8353 - val_loss: 0.5157 - val_accuracy: 0.8245Epoch 15/508000/8000 - 2s - loss: 0.4757 - accuracy: 0.8397 - val_loss: 0.5299 - val_accuracy: 0.8060
This time, the training gets terminated at Epoch 15 as there are 3 epochs with no improvement on validation accuracy (It has to be ≥ 0.001 to count as an improvement).
For more about early stopping, please check out this article:
towardsdatascience.com
CSVLogger is a callback that streams epoch results to a CSV file. First, let’s import it and create a CSVLogger object:
from tensorflow.keras.callbacks import CSVLoggercsv_log = CSVLogger("results.csv")
Next, we just need to pass the csv_log object to model.fit() method.
history_csv_logger = model.fit(    X_train,     y_train,     epochs=10,     validation_split=0.20,     batch_size=64,     verbose=2,    callbacks=[csv_log])
Once the training is completed, we can view the information in the CSV file
import pandas as pdpd.read_csv("results.csv", index_col='epoch')
Apart from the compulsory argument filename, the other 2 options separator and append are likely to be used quite often.
CSVLogger(filename, separator=',', append=False)
filename : is the filename of the CSV file.
separator: string used to separate elements in the CSV file.
append: is boolean and default to False, True append if file exists (useful for continuing training). False: overwrite existing file.
ModelCheckpoint is a callback to save the Keras model or model weight during training, so the model or weights can be loaded later to continue the training from the state saved.
First, let’s import it and create a ModelCheckpoint object:
from tensorflow.keras.callbacks import ModelCheckpointcheckpoint_path = 'model_checkpoints/'checkpoint = ModelCheckpoint(    filepath=checkpoint_path,    save_freq='epoch',    save_weights_only=True,    verbose=1)
Next, let’s pass the checkpoint object to model.fit() method for training.
history_checkpoint = model.fit(    X_train,     y_train,     epochs=10,     validation_split=0.20,     batch_size=64,     verbose=2,    callbacks=[checkpoint])
We should be able to see the above printout during the training.
Once the training is completed, we can get the test accuracy by running:
>>> get_test_accuracy(model, X_test, y_test)accuracy: 0.779
Let’s create a new model new_model to demonstrate how loading weights work. And by running get_test_accuracy(new_model, X_test, y_test), we get the test accuracy 0.086 for a model without loading any trained weights.
# Create a new model>>> new_model = create_model()# Without loading weight>>> get_test_accuracy(new_model, X_test, y_test)accuracy: 0.086
Next, let’s load weights with load_weights('model_checkpoints/') and get its test accuracy again. This time, we should be able to see the same accuracy as we trained the model model .
# Load weights>>> new_model.load_weights('model_checkpoints/')>>> get_test_accuracy(new_model, X_test, y_test)accuracy: 0.779
Below are the commonly used arguments you should know when using ModelCheckpoint callback
filepath: string or PathLike, the path to save the model file. filepath can contain named formatting options, for example, if filepath is weights.{epoch:02d}-{val_loss:.2f}, then the model checkpoints will be saved with the epoch number and the validation loss in the filename.
save_freq: 'epoch' or integer. When using 'epoch', the callback saves the model after each epoch. When using integer, the callback saves the model at end of this many batches.
save_weights_only: if True, then only the model’s weights will be saved (model.save_weights(filepath)), else the full model is saved (model.save(filepath)).
ReduceLROnPlateau is a callback to reduce the learning rate when a metric has stopped improving. This callback monitors a quantity and if no improvement is seen for a patience number of epochs, the learning rate is reduced by factor value (new_lr = lr * factor). Let’s see how this works with the help of an example.
First, let’s import it and create a ReduceLROnPlateau object:
from tensorflow.keras.callbacks import ReduceLROnPlateaureduce_lr = ReduceLROnPlateau(    monitor='val_loss',     factor=0.2,       patience=2,     min_lr=0.001,    verbose=2)
monitor='val_loss' to use validation loss as performance measure to reduce the learning rate. patience=2 means the learning rate is reduced as soon as 2 epochs with no improvement. min_delta=0.001 means the validation loss has to improve by at least 0.001 for it to count as an improvement. factor=0.2 means the new learning rate will be reduced as new_lr = lr * factor.
Let’s train the model with the reduce_lr callback
history_reduce_lr = model.fit(    X_train,     y_train,     epochs=50,     validation_split=0.20,     batch_size=64,     verbose=2,    callbacks=[reduce_lr])
You should get an output like above. In the above output, the ReduceLROnPlateau callback has been triggered at Epoch 30 and 37.
Let’s plot the learning rate and loss to get a clear picture.
Below are the commonly used arguments you should know when using ReduceLROnPlateau callback
monitor='val_loss': to use validation loss as performance measure to reduce the learning rate.
factor: the factor by which the learning rate will be reduced. new_lr = lr * factor.
patience: is the number of epochs with no improvement.
min_delta: Minimum change in the monitored quantity to qualify as an improvement, i.e. an absolute change of less than min_delta, will count as no improvement.
mode='auto': Should be one of auto, min or max. In 'min' mode, the learning rate will be reduced when the quantity monitored has stopped decreasing; in 'max' mode learning rate will be reduced when the quantity monitored has stopped increasing; in 'auto' mode, the direction is automatically inferred from the name of the monitored quantity.
min_lr: lower bound on the learning rate.
When training a neural network, it is often useful to reduce the learning rate as the training progresses. This can be done by using learning rate schedules or adaptive learning rate. LearningRateScheduler is a built-in callback for learning rate schedules.
For more details about learning rate schedules, please check out the following article:
towardsdatascience.com
Let’s see how LearningRateScheduler works with the help of an example.
First, let’s import it and create a schedule function:
from tensorflow.keras.callbacks import LearningRateSchedulerdef lr_decay(epoch, lr):    if epoch != 0 and epoch % 5 == 0:        return lr * 0.2    return lr
lr_decay() takes 2 arguments epoch (current epoch) &lr( current learning rate), and returns a new learning rate. Ourlr_decay() function will reduce the learning rate by a factor of 0.2 in every 5 epochs.
Let’s train the model with the reduce_lr callback
history_lr_schedule = model.fit(    X_train,     y_train,     epochs=20,     validation_split=0.20,     batch_size=64,     verbose=2,    callbacks=[LearningRateScheduler(lr_decay, verbose=1)])
You should get an output like above. And below is the plot of the learning rate.
Another useful callback is LambdaCallback. It is similar to Callback and allows us to build custom callbacks on-the-fly.
LambdaCallback is constructed with the following anonymous functions that will be called at the appropriate time.
on_epoch_begin: called at the beginning of every epoch.
on_epoch_end: called at the end of every epoch.
on_batch_begin: called at the beginning of every batch.
on_batch_end: called at the end of every batch.
on_train_begin: called at the beginning of model training.
on_train_end: called at the end of model training.
Note that the callbacks expect positional arguments, as:
on_epoch_begin and on_epoch_end expect two positional arguments: epoch, logs
on_batch_begin and on_batch_end expect two positional arguments: batch, logs
on_train_begin and on_train_end expect one positional argument: logs
Let’s see how LambdaCallback works with the help of an example.
First, let’s import it and create 3 different LambdaCallback:
from tensorflow.keras.callbacks import LambdaCallbackepoch_callback = LambdaCallback(    on_epoch_begin=lambda epoch,logs: print('Starting Epoch {}!'.format(epoch+1)))batch_loss_callback = LambdaCallback(    on_batch_end=lambda batch,logs: print('\n After batch {}, the loss is {:7.2f}.'.format(batch, logs['loss'])))train_finish_callback = LambdaCallback(    on_train_end=lambda logs: print('Training finished!'))
Let’s train the model with above callbacks
history_lambda_callback = model.fit(    X_train,     y_train,    epochs=2,                  # change epoch to 2 for demo purpose     validation_split=0.20,     batch_size=2000,           # change to 2000 for demo purpose    verbose=False,    callbacks=[epoch_callback, batch_loss_callback, train_finish_callback])
You should get an output like above. This is very useful
Along with the above functions, there are other callbacks that you might encounter or want to use in your Deep Learning project:
Callback: this is the base class used to build custom callback. It is similar to LambdaCallback but much powerful. We will have another discussion for it.
TensorBoard: a callback writes a log for TensorBoard, which is TensorFlow’s excellent visualization tool.
RemoteMonitor: a callback used to stream events to a server.
Thanks for reading. This article has covered the most popular Keras Callbacks.
Please check out the notebook for the source code.
Stay tuned if you are interested in the practical aspect of machine learning.
Learning Rate Schedules in practice
The Google’s 7 steps of Machine Learning in practice: a TensorFlow example for structured data
3 ways to create a Machine Learning Model with Keras and TensorFlow 2.0
Batch normalization in practice: an example with Keras and TensorFlow 2.0
Early stopping in Practice: an example with Keras and TensorFlow
More can be found from my Github