Tensorflow 2: Model validation, regularization, and callbacks
By: Rahul Bhadani
Tags: Machine Learning, Python, TensorFlow, Tensorflow2, Data Science

A common problem while developing a machine-learning model is overfitting. Besides, generalizing a model is a practical requirement. Otherwise, what’s the use of developing a machine-learning model if you cannot use it to make a successful prediction beyond the data a model was trained on. Generalization and overfitting avoidance are achieved through model selection and regularization. Tensorflow 2.x comes provides callbacks functionality through which a programmer can monitor the performance of a model on a held-out validation set while the training takes place.

Several approaches can be taken to make sure that the model you train doesn’t overfit, and generalizes well. A few of such approaches are:

Creating a validation set Regularization Early stopping

We will go through them one-by-one and understand those approaches can be incorporated into your training procedure.

Validation Set

A validation set helps to evaluate a model while it is being trained. From the dataset that is used for training, a portion of the set is kept a side that is never used for training but the performance of the model is evaluated using the validation set during the training procedure. Let's look at an example

import tensorflow as tf

import numpy as np

import pandas as pd

print(tf.__version__) from tensorflow.keras.models import Sequential

from tensorflow.keras.layers import Dense

from tensorflow.keras.optimizers import Adam

from tensorflow.keras.layers import Dense, Flatten, Softmax, Conv2D, MaxPooling2D # Load the Fashion-MNIST dataset

fashion_mnist_data = tf.keras.datasets.fashion_mnist

(train_images, train_labels), (test_images, test_labels) = fashion_mnist_data.load_data() labels = [

'T-shirt/top',

'Trouser',

'Pullover',

'Dress',

'Coat',

'Sandal',

'Shirt',

'Sneaker',

'Bag',

'Ankle boot'

] train_images = train_images/255.0

test_images = test_images/255.0 model = Sequential([

Conv2D(16, (3,3), activation = 'relu', input_shape=(28,28,1)),

MaxPooling2D((3,3)),

Flatten(),

Dense(10, activation='softmax')

]) opt = tf.keras.optimizers.Adam(learning_rate=0.005)

acc = tf.keras.metrics.SparseCategoricalAccuracy()

mae = tf.keras.metrics.MeanAbsoluteError() model.compile(optimizer=opt, loss = 'sparse_categorical_crossentropy',

metrics = [acc, mae]

) history = model.fit(train_images[..., np.newaxis], train_labels, epochs=10, batch_size=256, validation_split=0.2)

In this case, 20% of training data inputs is used for the performance evaluation of the model being trained.

Regularization

Now, we will include regularization into our model to prevent overfitting. In this article, we will use L2 weight regularization which adds squared magnitude as a penalty to the loss function. This is also known as ridge regression. In our example, we add an optional kernel regularizer option to the convolutional layer. Here, just to remind everyone, weight matrices are also called the kernel, hence the name ‘kernel regularizer’. We can also add a bias regularizer in the same way as we add a regularizer for weights. We can also add a dropout layer which has the same effect as that of a regularizer. To add a dropout layer, we can add a layer by calling Dropout(0.5) .

model = Sequential([

Conv2D(16, (3,3), activation = 'relu', input_shape=(28,28,1),

kernel_regularizer = tf.keras.regularizers.l2(0.01),

bias_regularizer = tf.keras.regularizers.l2(0.001)), MaxPooling2D((3,3)),

Flatten(),

Dense(10, activation='softmax')

]) opt = tf.keras.optimizers.Adam(learning_rate=0.005)

acc = tf.keras.metrics.SparseCategoricalAccuracy()

mae = tf.keras.metrics.MeanAbsoluteError() model.compile(optimizer=opt, loss = 'sparse_categorical_crossentropy',

metrics = [acc, mae]

)

Callbacks in Tensorflow

Callbacks are an important type of object in Tensorflow and Keras that are designed to monitor the loss in metrics at certain time points in a training run and perform some specified action that might depend on those losses in metric values. In TensorFlow 2, callbacks are stored in tensorflow.keras.callbacks module. The module provides a base class callback using which we can create a custom callback. The base class provides a number of methods that can be overridden in the inherited class. Let’s see an example:

from tensorflow.keras.callbacks import Callback

class trainingcallback(Callback):

def on_train_begin(self, logs = None):

# Do something at beginning of training

print("Starting training ...")



def on_train_batch_begin(self, batch, logs = None):

# Do something at the beginning of every batch iteration

print("Starting the training of batch {}".format(batch))



def on_epoch_begin(self, epoch, logs=None):

# Do something at the beginning of every epoch

print("Starting epoch {}".format(epoch))



def on_train_batch_end(self, batch, logs = None):

# Do something at the end of every batch iteration

print("Finished the training of batch {}".format(batch))



def on_epoch_end(self, epoch, logs=None):

# Do something at the end of every epoch

print("Finishing epoch {}".format(epoch))



def on_train_end(self, logs = None):

# Do something at beginning of training

print("Finished training.") model.compile(optimizer = "adam", loss = "mse")

history = model.fit(train_images[..., np.newaxis], train_labels, epochs=10, batch_size=256, validation_split=0.2, callbacks = [trainingcallback()])

You will be able to see that printout from callback functions as a specific event occur. Similar to training, we have test callbacks, and prediction callbacks — just replace the word train with test or predict.

Early stopping using callbacks

We can use Keras’s predefined EarlyStopping to implement early stopping in Keras using TensorFlow 2.

from tensorflow.keras.callbacks import EarlyStopping

es = EarlyStopping(monitor='val_accuracy', patience = 5, min_delta =0.01, metrics = ['accuracy'] )

history = model.fit(train_images[..., np.newaxis], train_labels, epochs=10, batch_size=256, validation_split=0.2, callbacks = [es()])

I hope the incorporation of regularization, batch normalization, and creating validation sets proves to be useful to readers who are enthusiastic about using TensorFlow 2 for their ML practice.

Notebook related to the article can be found at https://github.com/rahulbhadani/medium.com/tree/master/12_08_2020

Please, feel free to leave comments if there is a nagging issue related to the article’s topic.