Machine Learning Model Regularization in Practice: an example with Keras and TensorFlow 2.0 | by B. Chen | Towards Data Science

In this article, we will focus on incorporating regularization into our machine learning model and look at an example of how we do this in practice with Keras and TensorFlow 2.0.
Turns out many people are afraid of Maths, especially beginners and people changing career paths to data science.
Let’s avoid Maths and try to explain regularization in a more gentle way. If you are looking for a complete explanation (with a lot of math expressions) of regularisation, you might find the following articles useful:
Regularization in Machine Learning
Regularization: an important concept in Machine Learning
Regularization is a technique to combat the overfitting issue in machine learning. Overfitting, also known as High Variance, refers to a model that learns the training data too well but fail to generalize to new data. For example, the green line below is an overfitted model and the black line represents a regularized model.
Overfitting can be diagnosed by plotting training and validation loss. For example, in the below graph, training error is shown in blue, validation error in red, both as a function of the number of training epochs. If the validation error increases while the training error steadily decreases, then a situation of overfitting may have occurred.
Once overfitting is diagnosed, it is time to try different regularization techniques and parameters to see if they help. The most widely used regularization techniques are:
L1 regularization adds “absolute value of magnitude” as penalty term to the loss function. Some people say L1 can help with compressing the model. But in practice, L1 regularization makes your model sparse, helps only a little bit. L2 regularization is just used much more often.
L2 regularization (also known as weight decay) adds “squared magnitude” as penalty term to the loss function and it is used much more often than L1.
Dropout is widely used regularization technique in deep learning. It randomly shuts down some neurons in each iteration.
Data Augmentation adds extra fake training data. More training also helps with reducing overfitting.
Early Stopping is another widely used regularization technique in deep learning. It stops training when generalization error increases.
In the following article, we are going to incorporate L2 regularization and Dropout to reduce overfitting of a neural network model.
In order to run this tutorial, you need to install
TensorFlow 2, numpy, pandas, sklean, matplotlib
They can all be installed directly vis PyPI and I strongly recommend to create a new Virtual Environment. For a tutorial on creating a Python virtual environment
Create Virtual Environment using “virtualenv” and add it to Jupyter Notebook
Create Virtual Environment using “conda” and add it to Jupyter Notebook
This is a step by step tutorial and all instructions are in this article. For source code, please check out my Github machine learning repo.
This tutorial uses the Anderson Iris flower (iris) dataset for demonstration. The dataset contains a set of 150 records under five attributes: sepal length, sepal width, petal length, petal width, and class (known as target from sklearn datasets).
First, let’s import the libraries and obtain iris dataset from scikit-learn library. You can also download it from the UCI Iris dataset.
import tensorflow as tfimport pandas as pdimport numpy as npimport matplotlib.pyplot as pltfrom sklearn.datasets import load_irisfrom sklearn.model_selection import train_test_splitiris = load_iris()
For the purpose of exploring data, let’s load data into a DataFrame
# Load data into a DataFramedf = pd.DataFrame(iris.data, columns=iris.feature_names)# Convert datatype to floatdf = df.astype(float)# append "target" and name it "label"df['label'] = iris.target# Use string label insteaddf['label'] = df.label.replace(dict(enumerate(iris.target_names)))
And the df should look like below:
We notice the label column is a categorical feature and will need to convert it to one-hot encoding. Otherwise, our machine learning algorithm won’t be able to directly take in that as input.
# label -> one-hot encodinglabel = pd.get_dummies(df['label'], prefix='label')df = pd.concat([df, label], axis=1)# drop old labeldf.drop(['label'], axis=1, inplace=True)
Now, the df should look like:
Next, let’s create X and y. Keras and TensorFlow 2.0 only take in Numpy array as inputs, so we will have to convert DataFrame back to Numpy array.
# Creating X and yX = df[['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']]# Convert DataFrame into np arrayX = np.asarray(X)y = df[['label_setosa', 'label_versicolor', 'label_virginica']]# Convert DataFrame into np arrayy = np.asarray(y)
Finally, let’s split the dataset into a training set (80%)and a test set (20%) using train_test_split() from sklearn library.
X_train, X_test, y_train, y_test = train_test_split(  X,  y,  test_size=0.20)
Great! our data is ready for building a Machine Learning model.
Before applying regularization, let’s build a neural network without regularization and take a look at the overfitting issue.
There are 3 ways to create a machine learning model with Keras and Tensorflow 2. Since we are building a simple fully connected Neural Network and for simplicity, let’s use the easiest way: Sequential Model with Sequential().
Let’s go ahead and create a function called create_model() to return a Sequential model.
from tensorflow.keras.models import Sequentialfrom tensorflow.keras.layers import Densedef create_model():     model = Sequential([        Dense(64, activation='relu', input_shape=(4,)),        Dense(128, activation='relu'),        Dense(128, activation='relu'),        Dense(128, activation='relu'),        Dense(64, activation='relu'),        Dense(64, activation='relu'),        Dense(64, activation='relu'),        Dense(3, activation='softmax')    ])    return model
Notices that
The first layer (also known as the input layer) has the input_shape to set the input size (4,)
The input layer has 64 units, followed by 3 dense layers, each with 128 units. Then there are further 3 dense layers, each with 64 units. All these layers use the ReLU activation function.
The output Dense layer has 3 units and the softmax activation function.
By running
model = create_model()model.summary()
Should print out the model summary.
In order to train a model, we first have to configure our model using compile() and pass the following arguments:
Use Adam (adam) optimization algorithm as the optimizer
Use categorical cross-entropy loss function (categorical_crossentropy) for our multiple-class classification problem
For simplicity, use accuracy as our evaluation metrics to evaluate the model during training and testing.
model.compile(    optimizer='adam',     loss='categorical_crossentropy',     metrics=['accuracy'])
After that, we can call model.fit() to fit our model to the training data.
history = model.fit(    X_train,     y_train,     epochs=200,     validation_split=0.25,     batch_size=40,     verbose=2)
If all runs smoothly, we should get an output like below
Train on 90 samples, validate on 30 samplesEpoch 1/20090/90 - 1s - loss: 1.0939 - accuracy: 0.4333 - val_loss: 1.0675 - val_accuracy: 0.5333Epoch 2/20090/90 - 0s - loss: 1.0553 - accuracy: 0.6556 - val_loss: 1.0160 - val_accuracy: 0.7000............Epoch 200/20090/90 - 0s - loss: 0.0624 - accuracy: 0.9778 - val_loss: 0.1874 - val_accuracy: 0.9333
Once training is complete, it’s time to see if the model is any good with Model Evaluation. The Model Evaluation typically involves
Plot the progress on loss and accuracy metricsTest our model against data that has never been used for training. This is where the test dataset X_test that we set aside earlier come to play.
Plot the progress on loss and accuracy metrics
Test our model against data that has never been used for training. This is where the test dataset X_test that we set aside earlier come to play.
Let’s create a function plot_metric() for plotting metrics.
%matplotlib inline%config InlineBackend.figure_format = 'svg'def plot_metric(history, metric):    train_metrics = history.history[metric]    val_metrics = history.history['val_'+metric]    epochs = range(1, len(train_metrics) + 1)    plt.plot(epochs, train_metrics)    plt.plot(epochs, val_metrics)    plt.title('Training and validation '+ metric)    plt.xlabel("Epochs")    plt.ylabel(metric)    plt.legend(["train_"+metric, 'val_'+metric])    plt.show()
By running plot_metric(history, 'accuracy') to plot the progress on accuracy.
By running plot_metric(history, 'loss') to plot the progress on loss.
From the above graph, we can see that the model has overfitted the training data, so it outperforms the validation set.
To evaluate the model on the test set
# Evaluate the model on the test setmodel.evaluate(X_test, y_test, verbose=2)
And we should get an output like below
30/1 - 0s - loss: 0.0137 - accuracy: 1.0000[0.01365612167865038, 1.0]
First, let’s import Dropout and L2 regularization from TensorFlow Keras package
from tensorflow.keras.layers import Dropoutfrom tensorflow.keras.regularizers import l2
Then, we create a function called create_regularized_model() and it will return a model similar to the one we built before. But, this time we will add L2 regularization and Dropout layers, so this function takes 2 arguments: a L2 regularization factor and a Dropout rate.
Let’s add L2 regularization in all layers except the output layer [1].
Let’s add Dropout layer between every two dense layers.
def create_regularized_model(factor, rate):    model = Sequential([        Dense(64, kernel_regularizer=l2(factor), activation="relu", input_shape=(4,)),        Dropout(rate),        Dense(128, kernel_regularizer=l2(factor), activation="relu"),        Dropout(rate),        Dense(128, kernel_regularizer=l2(factor), activation="relu"),        Dropout(rate),        Dense(128, kernel_regularizer=l2(factor), activation="relu"),        Dropout(rate),        Dense(64, kernel_regularizer=l2(factor), activation="relu"),        Dropout(rate),        Dense(64, kernel_regularizer=l2(factor), activation="relu"),        Dropout(rate),        Dense(64, kernel_regularizer=l2(factor), activation="relu"),        Dropout(rate),        Dense(3, activation='softmax')    ])    return model
Let’s create the model with arguments L2 factor 0.0001 and Dropout rate 0.3
model = create_regularized_model(1e-5, 0.3)model.summary()
The regularized model can be trained just like the first model we built.
# First configure model using model.compile()model.compile(    optimizer='adam',     loss='categorical_crossentropy',     metrics=['accuracy'])# Then, train the model with fit()history = model.fit(    X_train,     y_train,     epochs=200,     validation_split=0.25,     batch_size=40,     verbose=2)
If all runs smoothly, we should get an output like below
Train on 90 samples, validate on 30 samplesEpoch 1/20090/90 - 2s - loss: 1.0855 - accuracy: 0.3333 - val_loss: 1.0873 - val_accuracy: 0.3000Epoch 2/20090/90 - 0s - loss: 1.0499 - accuracy: 0.3778 - val_loss: 1.0773 - val_accuracy: 0.3000............Epoch 200/20090/90 - 0s - loss: 0.1073 - accuracy: 0.9556 - val_loss: 0.1766 - val_accuracy: 0.9000
Now, let’s plot the progress on loss
plot_metric(history, 'loss')
From the graph, we can see that the overfitting it not completely fixed, but there is a significant improvement when we compare it to the unregularized model.
And finally, to evaluate the model on the test set
model.evaluate(X_test, y_test, verbose=2)
Should output something like
30/1 - 0s - loss: 0.0602 - accuracy: 0.9667[0.06016349419951439, 0.96666664]
Thanks for reading.
Please checkout the notebook on my Github for the source code.
Stay tuned if you are interested in the practical aspect of machine learning.
[1] Keras Layer weight regularizers