How To Implement Custom Regularization in TensorFlow(Keras) | by Richmond Alake | Towards Data Science

This article introduces the topic of regularization within neural networks. It includes details on how machine learning engineers and data scientist can go about implementing custom regularization techniques.
Regularization techniques reduce the possibility of a neural network overfitting by constraining the range of values that the weight values within the network hold.
The article below presents conventional regularization techniques and how they are implemented within TensorFlow(Keras). The referenced article below is aimed at readers who prefer an in-depth overview on the topic of regularization within neural networks.
towardsdatascience.com
Before we proceed further, it might be worth noting that in most scenarios, you will not be required to implement your custom regularization techniques.
Popular machine learning libraries such as TensorFlow, Keras and PyTorch have standard regularization techniques implemented within them.
The regularization technique I’m going to be implementing is the L2 regularization technique.
L2 regularization penalizes weight values. For both small weight values and relatively large ones, L2 regularization transforms the values to a number close to 0, but not quite 0.
L2 penalizes the sum of the square of the weights (weight2), therefore we will be implementing this logic in a python function.
def custom_l2_regularizer(weights):    return tf.reduce_sum(0.02 * tf.square(weights))
The code above is our custom L2 regularization technique.
Using TensorFlow’s mathematical operations we can calculate the sum of the square of the weights passed into the function.
tf.reduce_sum: Math operation that provides as its result, the sum of the values passed into the function.
tf.square: Math operation that squares the values within its arguments
And that’s all there is to it.
To utilize our custom regularizer, we pass it into a neural network layer, as shown below:
keras.layers.Dense(200, activation='relu', kernel_regularizer=custom_l2_regularizer),
Keas neural network layers can accept within its ‘kernel_regularizer’ argument custom-defined functions.
To see how well the implemented custom regularization technique works, we will build a simple neural network and perform the trivial task of image classification.
This section will include snippets of code that perform some common neural network implementation tasks, and also a GitHub repo link is available here that includes all code within this article.
To implement our neural network and load datasets, we will be utilizing the following tools and libraries: Keras, TensorFlow and NumPy.
import tensorflow as tffrom tensorflow import kerasimport numpy as np
Load and partition dataset into testing, validation and training.
It is also required that the pixel intensity of the images within the dataset are normalized from the value range 0–255 to 0–1.
An array ‘class_names’ is initialized to hold the item labels of the pieces of clothing represented in the images within the dataset.
(train_images, train_labels),(test_images, test_labels) = keras.datasets.fashion_mnist.load_data()train_images = train_images / 255.0test_images = test_images / 255.0validation_images = train_images[:5000]validation_labels = train_labels[:5000]class_names = ["T-shirt/top", "Trouser", "Pullover", "Dress", "Coat", "Sandal", "Shirt", "Sneaker", "Bag", "Ankle boot"]
Here we define the custom regularizer as explained earlier.
def custom_l2_regularizer(weights):    return tf.reduce_sum(0.02 * tf.square(weights))
Next step is to implement our neural network and its layers. An assignment of the appropriate parameters to each layer takes place here, including our custom regularizer.
model = keras.models.Sequential([    keras.layers.Flatten(input_shape=[28,28]),    keras.layers.Dense(200, activation='relu', kernel_regularizer=custom_l2_regularizer),    keras.layers.Dense(100, activation='relu', kernel_regularizer=custom_l2_regularizer),    keras.layers.Dense(50, activation='relu', kernel_regularizer=custom_l2_regularizer),    keras.layers.Dense(10, activation='softmax')])
To ensure that our neural network trains properly, we utilize the stochastic gradient descent optimization algorithm and set some arbitrary hyperparameters such as epochs and learning rates.
sgd = keras.optimizers.SGD(lr=0.01)model.compile(loss="sparse_categorical_crossentropy", optimizer=sgd, metrics=["accuracy"])model.fit(train_images, train_labels, epochs=60, validation_data=(validation_images, validation_labels))
After training, an evaluation of the trained model is conducted on the test dataset. The evaluation data must contain data that the model has not encountered during training.
model.evaluate(test_images, test_labels)
To verify the accuracy of our model, we can take a subset of our test images and run inference to verify their results.
practical_test_images =  test_images[:10]predictions = model.predict_classes(practical_test_images)print(predictions)print(np.array(class_names)[predictions])
Implementing custom components of neural networks such as regularizers is very easy with Keras; you could even go further and explore how to implement custom metrics, activation functions, loss functions and plenty more with Keras.
To connect with me or find more content similar to this article, do the following:
Subscribe to my YouTube channel for video contents coming soon hereFollow me on MediumConnect and reach me on LinkedIn
Subscribe to my YouTube channel for video contents coming soon here
Follow me on Medium
Connect and reach me on LinkedIn