Regularization Techniques And Their Implementation In TensorFlow(Keras) | by Richmond Alake | Towards Data Science

Deep Neural Networks(DNN) have a vast amount of weights parameters internal to the architecture that learn a range of values. These range of values are the essential key to enabling the neural network to solve huge complex functions.
The deeper a neural network is, the more representational power it possesses, but, there is a shortcoming that occurs as the number of weight parameter increases. This shortcoming is that the neural network is more prone to overfitting the training dataset.
Overfitting: This problem involves the algorithm predicting new instances of patterns presented to it, based too closely on instances of patterns it observed and learnt during training. This can cause the machine-learning algorithm to not generalize accurately to unseen data. Overfitting can occur if the training data does not accurately represent the distribution of test data. Overfitting can be fixed by reducing the number of features in the training data and reducing the complexity of the network through various techniques
Regularization techniques reduce the possibility of a neural network overfitting by constraining the range of values that the weight values within the network hold(more on this later).
This article introduces two regularization strategies that impose constraining terms on the results of loss functions.
The loss function is a method that quantifies ‘how well’ a machine learning model performs. The quantification is an output(cost) based on a set of inputs, which are referred to as parameter values. The parameter values are used to estimate a prediction, and the ‘loss’ is the difference between the predictions and the actual values.
This article won’t focus on the maths of regularization. Instead, this article presents some standard regularization methods and how to implement them within neural networks using TensorFlow(Keras).
For more details on the maths, these article by Raimi Karim and Renu Khandelwal present L1 and L2 regularization maths reasonably.
medium.com
towardsdatascience.com
From the previous section, we can understand that the regularization technique acts on the weights parameters within neural networks. More specifically, it modifies the result loss function, which in turn modifies the weight values produced.
L1 regularization effect on the neural network weight values is that it penalizes weight values that are close to 0 by making them equal to 0. Negative weight values also take a value of 0; so if a weight value is -2, under the effect of L1 regularization, it becomes 0.
The general intuition with L1 regularization is that, if a weight value is close to 0 or very small, then it’s negligible when it comes to the overall performance of the model, therefore making it 0 does not affect the model’s performance and can reduce the memory capacity of the model.
L1 penalizes the sum of the absolute values of the weights (|weight|)
I know I said I wouldn’t get into the maths, but the mathematical notation below should be relatively easy to understand.
We have our loss function, in this case, Mean squared error.
Then we add the product of the sum of the absolute values of the weights and the regularization hyperparameter value, represented by the lambda symbol ().
‘i’ in the mathematical notation represent the index of the current weight, and ’n’ represents the total number of weights values within the layer. ‘W’ represents the weight value.
L2 regularization also penalizes weight values. For both small weight values and relatively large ones, L2 regularization transforms the values to a number close to 0, but not quite 0.
l2 penalizes the sum of the square of the weights (weight2)
If you were to combine the effect of L1 and L2 regularization techniques, then you would arrive at ‘Elastic Net regularization’.
Regularization techniques effect is imposed on the neural network during training and not inference.
Now we have some basic understanding of regularization(feel free to explore the maths of both methods) and some examples, let’s see how they are implemented.
The first step is to import tools and libraries that are utilized to either implement or support the implementation of the neural network.
TensorFlow: An open-source platform for the implementation, training, and deployment of machine learning models.
Keras: An open-source library used for the implementation of neural network architectures that run on both CPUs and GPUs.
import tensorflow as tffrom tensorflow import keras
The dataset we’ll be utilizing is the trivial fashion-MNIST dataset.
The fashion-MNIST dataset contains 70,000 images of clothing. More specifically, it includes 60,000 training examples and 10,000 testing examples, that are all grayscale images with dimension 28 x 28 categorized into ten classes.
Preparation of the dataset includes the normalization of the training image and test images by dividing each pixel value by 255.0. This places the pixel value within the range 0 and 1.
A validation portion of the dataset is also created at this stage. This group of the dataset is utilized during training to assess the performance of the network at various iterations.
(train_images, train_labels),(test_images, test_labels) = keras.datasets.fashion_mnist.load_data()train_images = train_images / 255.0test_images = test_images / 255.0validation_images = train_images[:5000]validation_labels = train_labels[:5000]
Next, we implement a simple model using the Keras Sequential API. The hidden layers in our model have a variety of regularization techniques used.
To add a regularizer to a layer, you simply have to pass in the prefered regularization technique to the layer’s keyword argument ‘kernel_regularizer’.
The Keras regularization implementation methods can provide a parameter that represents the regularization hyperparameter value. This is shown in some of the layers below.
Keras provides an implementation of the l1 and l2 regularizers that we will utilize in some of the hidden layers in the code snippet below. Also, we include a layer that leverages both l1 and l2 regularization.
And that’s all there is to implementing various regularization techniques within neural networks. Not too difficult.
model = keras.models.Sequential([    keras.layers.Flatten(input_shape=[28,28]),    keras.layers.Dense(200, activation='relu', kernel_regularizer=keras.regularizers.l1()),    keras.layers.Dense(100, activation='relu', kernel_regularizer=keras.regularizers.l2(0.001)),    keras.layers.Dense(50, activation='relu', kernel_regularizer=keras.regularizers.l1_l2(0.01)),    keras.layers.Dense(10, activation='softmax')])
In the next snippet of code we set and specify the optimization algorithm used to train the implemented neural network; along with the loss function and hyperparameters such as learning rate and the number of epochs.
sgd = keras.optimizers.SGD(lr=0.01)model.compile(loss="sparse_categorical_crossentropy", optimizer=sgd, metrics=["accuracy"])model.fit(train_images, train_labels, epochs=60, validation_data=(validation_images, validation_labels))
The evaluation of the model performance is conducted using the test data set aside earlier.
With evaluation results, you can decide to fine-tune the network hyperparameters or move forward to production after observing the accuracy of the evaluation over the test dataset.
model.evaluate(test_images, test_labels)
The implemented model will probably have a better performance on the test dataset if the regularization terms on the loss function were excluded from the layers and trained for the same number of epochs.
Regularization is more commonly utilized in a deeper neural network with millions of parameters and more features.
I hope you the reader now have an intuition on the variety of regularization techniques and how they can be implemented.
It might be of interest to explore other regularization methods within neural network implementation and training, such as Dropout or Early Stopping.
Below is the GitHub repository for the code included in this article.