A Practical Introduction to Keras Callbacks in TensorFlow 2
By: B. Chen
Tags: TensorFlow, Keras, Machine Learning, Callback, Deep Learning

1. EarlyStopping

EarlyStopping is a built-in callback designed for early stopping. First, let’s import it and create an early stopping object:

from tensorflow.keras.callbacks import EarlyStopping early_stopping = EarlyStopping()

EarlyStopping() has a few options and by default:

monitor='val_loss' : to use validation loss as performance measure to terminate the training.

: to use validation loss as performance measure to terminate the training. patience=0 : is the number of epochs with no improvement. The value 0 means the training is terminated as soon as the performance measure gets worse from one epoch to the next.

Next, we just need to pass the callback object to model.fit() method.

history = model.fit(

X_train,

y_train,

epochs=50,

validation_split=0.20,

batch_size=64,

verbose=2,

callbacks=[early_stopping]

)

You can see that early_stopping get passed in a list to the callbacks argument. It is a list because in practice we might be passing a number of callbacks for performing different tasks, for example, debugging and learning rate schedules.

By executing the statement, you should get an output like below:

Train on 8000 samples, validate on 2000 samples

Epoch 1/50

8000/8000 - 6s - loss: 1.5632 - accuracy: 0.5504 - val_loss: 1.1315 - val_accuracy: 0.6605

......

......

Epoch 10/50

8000/8000 - 2s - loss: 0.5283 - accuracy: 0.8213 - val_loss: 0.5539 - val_accuracy: 0.8170

Epoch 11/50

8000/8000 - 2s - loss: 0.5141 - accuracy: 0.8281 - val_loss: 0.5644 - val_accuracy: 0.7990

The training gets terminated at Epoch 11 due to the increase of val_loss value and that is exactly the conditions monitor='val_loss' and patience=0 .

It’s often more convenient to look at a plot, let’s run plot_metric(history, 'loss') to get a clear picture. In the below graph, validation loss is shown in orange and it’s clear that validation error increases at Epoch 11.

Image made by author (Please check out notebook)

Arguments

Apart from the options monitor and patience we mentioned early, the other 2 options min_delta and mode are likely to be used quite often.

EarlyStopping(

monitor='val_loss',

patience=0,

min_delta=0,

mode='auto'

)

monitor='val_loss' : to use validation loss as performance measure to terminate the training.

: to use validation loss as performance measure to terminate the training. patience=0 : is the number of epochs with no improvement. The value 0 means the training is terminated as soon as the performance measure gets worse from one epoch to the next.

: is the number of epochs with no improvement. The value means the training is terminated as soon as the performance measure gets worse from one epoch to the next. min_delta : Minimum change in the monitored quantity to qualify as an improvement, i.e. an absolute change of less than min_delta , will count as no improvement.

: Minimum change in the monitored quantity to qualify as an improvement, i.e. an absolute change of less than , will count as no improvement. mode='auto' : Should be one of auto , min or max . In 'min' mode, training will stop when the quantity monitored has stopped decreasing; in 'max' mode it will stop when the quantity monitored has stopped increasing; in 'auto' mode, the direction is automatically inferred from the name of the monitored quantity.

And here is an example of a customized early stopping:

custom_early_stopping = EarlyStopping(

monitor='val_accuracy',

patience=3,

min_delta=0.001,

mode='max'

)

monitor='val_accuracy' to use validation accuracy as performance measure to terminate the training. patience=3 means the training is terminated as soon as 3 epochs with no improvement. min_delta=0.001 means the validation accuracy has to improve by at least 0.001 for it to count as an improvement. mode='max' means it will stop when the quantity monitored has stopped increasing.

Let’s go ahead, run it with the customized early stopping, and plot the accuracy.

Train on 8000 samples, validate on 2000 samples

Epoch 1/50

......

Epoch 12/50

8000/8000 - 2s - loss: 0.5043 - accuracy: 0.8290 - val_loss: 0.5311 - val_accuracy: 0.8250

Epoch 13/50

8000/8000 - 3s - loss: 0.4936 - accuracy: 0.8332 - val_loss: 0.5310 - val_accuracy: 0.8155

Epoch 14/50

8000/8000 - 2s - loss: 0.4835 - accuracy: 0.8353 - val_loss: 0.5157 - val_accuracy: 0.8245

Epoch 15/50

8000/8000 - 2s - loss: 0.4757 - accuracy: 0.8397 - val_loss: 0.5299 - val_accuracy: 0.8060

Image made by author (Please check out notebook)

This time, the training gets terminated at Epoch 15 as there are 3 epochs with no improvement on validation accuracy (It has to be ≥ 0.001 to count as an improvement).

For more about early stopping, please check out this article:

2. CSVLogger

CSVLogger is a callback that streams epoch results to a CSV file. First, let’s import it and create a CSVLogger object:

from tensorflow.keras.callbacks import CSVLogger csv_log = CSVLogger("results.csv")

Next, we just need to pass the csv_log object to model.fit() method.

history_csv_logger = model.fit(

X_train,

y_train,

epochs=10,

validation_split=0.20,

batch_size=64,

verbose=2,

callbacks=[csv_log]

)

Once the training is completed, we can view the information in the CSV file

import pandas as pd pd.read_csv("results.csv", index_col='epoch')

Image made by author (Please check out notebook)

Arguments

Apart from the compulsory argument filename , the other 2 options separator and append are likely to be used quite often.

CSVLogger(filename, separator=',', append=False)

filename : is the filename of the CSV file.

: is the filename of the CSV file. separator : string used to separate elements in the CSV file.

: string used to separate elements in the CSV file. append : is boolean and default to False , True append if file exists (useful for continuing training). False : overwrite existing file.

3. ModelCheckpoint

ModelCheckpoint is a callback to save the Keras model or model weight during training, so the model or weights can be loaded later to continue the training from the state saved.

First, let’s import it and create a ModelCheckpoint object:

from tensorflow.keras.callbacks import ModelCheckpoint checkpoint_path = 'model_checkpoints/' checkpoint = ModelCheckpoint(

filepath=checkpoint_path,

save_freq='epoch',

save_weights_only=True,

verbose=1

)

Next, let’s pass the checkpoint object to model.fit() method for training.

history_checkpoint = model.fit(

X_train,

y_train,

epochs=10,

validation_split=0.20,

batch_size=64,

verbose=2,

callbacks=[checkpoint]

)

Image made by author (Please check out notebook)

We should be able to see the above printout during the training.

Once the training is completed, we can get the test accuracy by running:

>>> get_test_accuracy(model, X_test, y_test)

accuracy: 0.779

Loading weights

Let’s create a new model new_model to demonstrate how loading weights work. And by running get_test_accuracy(new_model, X_test, y_test) , we get the test accuracy 0.086 for a model without loading any trained weights.

# Create a new model

>>> new_model = create_model() # Without loading weight

>>> get_test_accuracy(new_model, X_test, y_test)

accuracy: 0.086

Next, let’s load weights with load_weights('model_checkpoints/') and get its test accuracy again. This time, we should be able to see the same accuracy as we trained the model model .

# Load weights

>>> new_model.load_weights('model_checkpoints/')

>>> get_test_accuracy(new_model, X_test, y_test)

accuracy: 0.779

Arguments

Below are the commonly used arguments you should know when using ModelCheckpoint callback

filepath : string or PathLike , the path to save the model file. filepath can contain named formatting options, for example, if filepath is weights.{epoch:02d}-{val_loss:.2f} , then the model checkpoints will be saved with the epoch number and the validation loss in the filename.

: string or , the path to save the model file. can contain named formatting options, for example, if is , then the model checkpoints will be saved with the epoch number and the validation loss in the filename. save_freq : 'epoch' or integer. When using 'epoch' , the callback saves the model after each epoch. When using integer, the callback saves the model at end of this many batches.

: or integer. When using , the callback saves the model after each epoch. When using integer, the callback saves the model at end of this many batches. save_weights_only : if True , then only the model’s weights will be saved ( model.save_weights(filepath) ), else the full model is saved ( model.save(filepath) ).

4. ReduceLROnPlateau

ReduceLROnPlateau is a callback to reduce the learning rate when a metric has stopped improving. This callback monitors a quantity and if no improvement is seen for a patience number of epochs, the learning rate is reduced by factor value ( new_lr = lr * factor ). Let’s see how this works with the help of an example.

First, let’s import it and create a ReduceLROnPlateau object:

from tensorflow.keras.callbacks import ReduceLROnPlateau reduce_lr = ReduceLROnPlateau(

monitor='val_loss',

factor=0.2,

patience=2,

min_lr=0.001,

verbose=2

)

monitor='val_loss' to use validation loss as performance measure to reduce the learning rate. patience=2 means the learning rate is reduced as soon as 2 epochs with no improvement. min_delta=0.001 means the validation loss has to improve by at least 0.001 for it to count as an improvement. factor=0.2 means the new learning rate will be reduced as new_lr = lr * factor .

Let’s train the model with the reduce_lr callback

history_reduce_lr = model.fit(

X_train,

y_train,

epochs=50,

validation_split=0.20,

batch_size=64,

verbose=2,

callbacks=[reduce_lr]

)

Image made by author (Please check out notebook)

You should get an output like above. In the above output, the ReduceLROnPlateau callback has been triggered at Epoch 30 and 37.

Let’s plot the learning rate and loss to get a clear picture.

Image made by author (Please check out notebook)

Image made by author (Please check out notebook)

Arguments

Below are the commonly used arguments you should know when using ReduceLROnPlateau callback

monitor='val_loss' : to use validation loss as performance measure to reduce the learning rate.

: to use validation loss as performance measure to reduce the learning rate. factor : the factor by which the learning rate will be reduced. new_lr = lr * factor .

: the factor by which the learning rate will be reduced. . patience : is the number of epochs with no improvement.

: is the number of epochs with no improvement. min_delta : Minimum change in the monitored quantity to qualify as an improvement, i.e. an absolute change of less than min_delta , will count as no improvement.

: Minimum change in the monitored quantity to qualify as an improvement, i.e. an absolute change of less than , will count as no improvement. mode='auto' : Should be one of auto , min or max . In 'min' mode, the learning rate will be reduced when the quantity monitored has stopped decreasing; in 'max' mode learning rate will be reduced when the quantity monitored has stopped increasing; in 'auto' mode, the direction is automatically inferred from the name of the monitored quantity.

: Should be one of , or . In mode, the learning rate will be reduced when the quantity monitored has stopped decreasing; in mode learning rate will be reduced when the quantity monitored has stopped increasing; in mode, the direction is automatically inferred from the name of the monitored quantity. min_lr : lower bound on the learning rate.

5. LearningRateScheduler

When training a neural network, it is often useful to reduce the learning rate as the training progresses. This can be done by using learning rate schedules or adaptive learning rate. LearningRateScheduler is a built-in callback for learning rate schedules.

For more details about learning rate schedules, please check out the following article:

Let’s see how LearningRateScheduler works with the help of an example.

First, let’s import it and create a schedule function:

from tensorflow.keras.callbacks import LearningRateScheduler def lr_decay(epoch, lr):

if epoch != 0 and epoch % 5 == 0:

return lr * 0.2

return lr

lr_decay() takes 2 arguments epoch (current epoch) & lr ( current learning rate), and returns a new learning rate. Our lr_decay() function will reduce the learning rate by a factor of 0.2 in every 5 epochs.

Let’s train the model with the reduce_lr callback

history_lr_schedule = model.fit(

X_train,

y_train,

epochs=20,

validation_split=0.20,

batch_size=64,

verbose=2,

callbacks=[LearningRateScheduler(lr_decay, verbose=1)]

)

You should get an output like above. And below is the plot of the learning rate.

Image made by author (Please check out notebook)

6. LambdaCallback

Another useful callback is LambdaCallback . It is similar to Callback and allows us to build custom callbacks on-the-fly.

LambdaCallback is constructed with the following anonymous functions that will be called at the appropriate time.

on_epoch_begin : called at the beginning of every epoch.

: called at the beginning of every epoch. on_epoch_end : called at the end of every epoch.

: called at the end of every epoch. on_batch_begin : called at the beginning of every batch.

: called at the beginning of every batch. on_batch_end : called at the end of every batch.

: called at the end of every batch. on_train_begin : called at the beginning of model training.

: called at the beginning of model training. on_train_end : called at the end of model training.

Note that the callbacks expect positional arguments, as:

on_epoch_begin and on_epoch_end expect two positional arguments: epoch , logs

and expect two positional arguments: , on_batch_begin and on_batch_end expect two positional arguments: batch , logs

and expect two positional arguments: , on_train_begin and on_train_end expect one positional argument: logs

Let’s see how LambdaCallback works with the help of an example.

First, let’s import it and create 3 different LambdaCallback :

from tensorflow.keras.callbacks import LambdaCallback epoch_callback = LambdaCallback(

on_epoch_begin=lambda epoch,logs: print('Starting Epoch {}!'.format(epoch+1))

) batch_loss_callback = LambdaCallback(

on_batch_end=lambda batch,logs: print('

After batch {}, the loss is {:7.2f}.'.format(batch, logs['loss']))

) train_finish_callback = LambdaCallback(

on_train_end=lambda logs: print('Training finished!')

)

Let’s train the model with above callbacks

history_lambda_callback = model.fit(

X_train,

y_train,

epochs=2, # change epoch to 2 for demo purpose

validation_split=0.20,

batch_size=2000, # change to 2000 for demo purpose

verbose=False,

callbacks=[epoch_callback, batch_loss_callback, train_finish_callback]

)

Image made by author (Please check out notebook)

You should get an output like above. This is very useful

Other callback functions

Along with the above functions, there are other callbacks that you might encounter or want to use in your Deep Learning project:

Callback : this is the base class used to build custom callback. It is similar to LambdaCallback but much powerful. We will have another discussion for it.

: this is the base class used to build custom callback. It is similar to but much powerful. We will have another discussion for it. TensorBoard : a callback writes a log for TensorBoard, which is TensorFlow’s excellent visualization tool.

: a callback writes a log for TensorBoard, which is TensorFlow’s excellent visualization tool. RemoteMonitor : a callback used to stream events to a server.

That’s it

Thanks for reading. This article has covered the most popular Keras Callbacks.

Please check out the notebook for the source code.

Stay tuned if you are interested in the practical aspect of machine learning.

You may be interested in some of my other TensorFlow articles:

More can be found from my Github