The Problem of Vanishing Gradients | by Animesh Agarwal | Towards Data Science

Vanishing gradients occur while training deep neural networks using gradient-based optimization methods. It occurs due to the nature of the backpropagation algorithm that is used to train the neural network.
Explain the problem of vanishing gradients: We will understand why the problem of vanishing gradients exists. We will also look at why this problem is more observed while using the sigmoid activation function and how RELU reduces the problem.A comparative study using TensorFlow: We will train an MLP classifier on the Fashion MNIST dataset using the sigmoid and RELU activation functions and compare the gradients.
Explain the problem of vanishing gradients: We will understand why the problem of vanishing gradients exists. We will also look at why this problem is more observed while using the sigmoid activation function and how RELU reduces the problem.
A comparative study using TensorFlow: We will train an MLP classifier on the Fashion MNIST dataset using the sigmoid and RELU activation functions and compare the gradients.
Before we can understand why the problem of vanishing gradients exists, we should have some understanding of the equations of the backpropagation algorithm. I will not derive the backpropagation equations as it is beyond the scope of this article. However, I have attached a screenshot from this famous book.
Observations:
From equation 1&2 we can see that the error in the output layer depends on the derivative of the activation function.From equation 2 we can observe that the error in the previous/hidden layer is proportional to the weights, the error from the outer layers and the derivative of the activation function. The error from outer layers is in turn dependent on the derivative of the activation function of its layer.From equation 1,2 &4 we can conclude that the gradients for weights for any layer are dependent on the product of derivatives of the activation layers.
From equation 1&2 we can see that the error in the output layer depends on the derivative of the activation function.
From equation 2 we can observe that the error in the previous/hidden layer is proportional to the weights, the error from the outer layers and the derivative of the activation function. The error from outer layers is in turn dependent on the derivative of the activation function of its layer.
From equation 1,2 &4 we can conclude that the gradients for weights for any layer are dependent on the product of derivatives of the activation layers.
Consider the graph of sigmoid function and it’s derivative. Observe that for very large values for sigmoid function the derivative takes a very low value. If the neural network has many hidden layers, the gradients in the earlier layers will become very low as we multiply the derivatives of each layer. As a result, learning in the earlier layers becomes very slow.
On the other hand, the derivative of the RELU function is either zero or 1. Even if we multiply the derivatives for many layers, there will be no degradation unlike the case of the sigmoid function (assuming that RELU is not operating in the dead-region). However, if you carefully look at equation 2, the error signal is also dependent on the weights of the network. If the weights of the network are constantly below zero, the gradients will diminish slowly.
Now that we have understood the problem of vanishing gradients, let’s train two different MLP Classifiers, one using the sigmoid and the other using the RELU activation functions.
Let’s get started.
Import Libraries
import tensorflow as tfimport numpy as npfrom tensorflow import kerasimport matplotlib.pyplot as plt from tqdm import tqdm
Download the dataset
We will be using the Fashion MNIST dataset. It consists of 70000 grayscale images of 10 different product categories. Each product category has 7000 samples. You can read more about the dataset here.
We will use Keras to download the Fashion MNIST dataset.
fashion_mnist = keras.datasets.fashion_mnist(x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()print(x_train.shape)print(y_train.shape)print(x_test.shape)print(y_test.shape)(60000,28,28)(60000,)(10000,28,28)(10000,)
As we will be using an MLP Classifier, we need to flatten the 28*28 grayscale image into a 784 long vector.
x_train = x_train.reshape(60000,-1)x_test = x_test.reshape(10000,-1)
The pixel values of the images are between 0 and 1. We will define a function to standardize the dataset by removing the mean and scaling to unit variance.
def normalize(x_train, x_test):  train_mean = np.mean(x_train)  train_std = np.mean(x_train)  x_train = (x_train - train_mean)/train_std  x_test = (x_test - train_mean)/train_std    return x_train, x_test
The product categories are labeled from 0 to 9. We will define a utility to convert the values into one hot encoding.
def convert_to_one_hot(labels):  no_samples = labels.shape[0]  n_classes = np.max(labels) + 1  one_hot = np.zeros((no_samples, n_classes))  one_hot[np.arange(no_samples),labels.ravel()] = 1  return one_hot
Next, we preprocess our dataset by passing the raw values to the utility functions we defined above.
x_train, x_test = normalize(x_train, x_test)y_train = convert_to_one_hot(y_train)y_test = convert_to_one_hot(y_test)
We will train an MLP classifier using TensorFlow. We will create a 6 layer network including the input and output layers. We will train two different models, one using the sigmoid activation function and the other using the RELU activation function. At every iteration of the backpropagation algorithm, we will store the gradients of loss with respect to the weights at each layer. Finally, through some visualizations, we will compare
The gradients of final layers vs the earlier layersThe gradients of sigmoid activation function vs the RELU activation function
The gradients of final layers vs the earlier layers
The gradients of sigmoid activation function vs the RELU activation function
Let’s build the model.
First, we define the placeholders for inputs and the targets.
def get_placeholders(input_size, output_size):  inputs = tf.placeholder(dtype=tf.float32, shape=[None, input_size], name="inputs")  targets = tf.placeholder(dtype=tf.float32, shape=[None, output_size], name="targets")  return inputs, targets
We will define a function to create a fully connected layer. The weights at each layer are named in the format “kernel/<layer_num>”. This will be useful later to extract the gradients. More on this later.
def dense_layer(input, hidden_units, layer_no, activation_fn= tf.nn.relu):  weights_name = "kernel/{}".format(layer_no)  bias_name = "biases/{}".format(layer_no)  weights = tf.get_variable(weights_name, shape=[input.shape[1], hidden_units], initializer = tf.contrib.layers.xavier_initializer(seed=0))  biases = tf.get_variable(bias_name, shape=[hidden_units], initializer = tf.zeros_initializer())  output = tf.add(tf.matmul(input, weights), biases)  if activation_fn:    return activation_fn(output)  else:    return output
Next, we define a function to create the forward pass of the network. We apply the activation function to each hidden layer. The output layer is not passed through an activation function as softmax activation for the output layer will be included in the loss definition.
def build_network(features, labels, hidden_units, num_layers, activation_fn):  inputs = features  for layer in range(num_layers-1):    inputs = dense_layer(inputs, hidden_units[layer], layer+1, activation_fn)  logits = dense_layer(inputs, 10, num_layers, None)   return logits
As we want to classify between 10 different product categories, we apply the softmax activation to the output layer to get the final probabilities. We will use cross-entropy loss as a measure to identify how good is our network. Tensorflow provides a handy utility to compute both of them together using tf.nn.softmax_cross_entropy_with_logits_v2.
def compute_loss(logits, labels):  loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(labels = labels , logits= logits))  train_op = tf.train.AdamOptimizer(0.001)  return loss, train_op
If you see carefully, we haven’t added the minimize operation on the optimizer. The minimize operation internally performs two operations. First, it computes the gradients of the loss w.r.t the weights and biases. Second, it updates the weights and biases using the gradients computed above.
As we want the value of the gradients at each iteration to study the problem of vanishing gradients, we will define two separate operations, one to compute the gradients and the second to apply the gradients. We will revisit this later when we train our model.
We will also define a utility to compute the accuracy of the model. tf.argmax(logits,1) will return the index for which the probability is maximum. As the labels are one-hot encoded, tf.argmax(labels,1) will return the index where the value is 1. We compare both these values to determine the correct predictions and accuracy.
def compute_accuracy(logits, labels):  correct_predictions = tf.equal(tf.argmax(logits,1), tf.argmax(labels,1))  accuracy = tf.reduce_mean(tf.cast(correct_predictions, tf.float32))  return accuracy
We will define a function to return the names of all the weights in the network. Remember that we defined the name of the weights as kernel/<layer_num>? Tensorflow internally appends “:0” to the name of the variable. You can read more about it here.
def get_weights_name(num_layers):  return ["kernel/{}:0".format(layer+1) for layer in range(num_layers)]
We will define a function to train the model. We will train our model with sigmoid and RELU activations.
def train_model(features, labels, hidden_units, epochs, batch_size, learning_rate, num_layers, activation_fn):  tf.reset_default_graph()  input_size = features.shape[1]  output_size = labels.shape[1]    # get the placeholders   inputs, targets = get_placeholders(input_size,output_size)    # create a dataset  dataset = tf.data.Dataset.from_tensor_slices((inputs, targets))    # make the required batches   dataset = dataset.batch(batch_size)    # initialize the iterator for the dataset   iterator = dataset.make_initializable_iterator()    # get the next batch  x_batch, y_batch = iterator.get_next()    # forward network  logits = build_network(x_batch, y_batch, hidden_units, num_layers, activation_fn)   # compute the loss  loss, train_op = compute_loss(logits, y_batch)    ''' instead of directly using the minimize function on the optimizer,    we break the operation into two parts   1. compute the gradients   2. apply the gradients  '''  grads_and_vars_tensor = train_op.compute_gradients(loss, tf.trainable_variables())  optimizer = train_op.apply_gradients(grads_and_vars_tensor)    # compute the accuracy of the model  accuracy = compute_accuracy(logits, y_batch)    init_op = tf.global_variables_initializer()  saver = tf.train.Saver()    with tf.Session() as sess:    sess.run(init_op)    def_graph = tf.get_default_graph()    gradient_means = []    losses = []    accuracies = []    train_samples = features.shape[0]        iteration = 0    # get the name of all the trainable variables    trainable_variables = [var.name for var in tf.trainable_variables()]    for epoch in range(epochs):      epoch_loss = 0      total_accuracy = 0      # run the iterator's initializer      sess.run(iterator.initializer, feed_dict={inputs:features, targets:labels})      try:                while True:          gradients = []          batch_loss, grads_and_vars,  _ , batch_accuracy = sess.run([loss, grads_and_vars_tensor, optimizer, accuracy])          if iteration % 100 == 0:                        # create a dictionary of all the trianbale variables and it's gradients            var_grads = dict(zip(trainable_variables, [grad for grad,var in grads_and_vars]))                        # get the gradients of all the weights for all the layers            weights_grads = [var_grads[var] for var in get_weights_name(num_layers)]                        # take the mean of the gradients at each layer            mean_values = [np.mean(np.abs(val)) for val in weights_grads]            gradient_means.append(mean_values)                      epoch_loss += batch_loss          total_accuracy += batch_accuracy*batch_size          iteration += 1      except tf.errors.OutOfRangeError:        pass      print("Total Iterations {}".format(iteration))      acc = total_accuracy/train_samples      accuracies.append(acc)      losses.append(epoch_loss)      print("Epoch: {}/{} , Loss: {} , Accuracy: {}".format(epoch, epochs, epoch_loss, acc))        return losses, accuracies, gradient_means
As the function is very big, we will break it down into smaller parts to understand.
Feeding the data to the model
First, we use Datasets and Iterators to feed the data to the network. If you want to read more in detail about Datasets and Iterators, you can refer to this blog. We then define the logits, loss, optimizer and accuracy operations using the routines we defined earlier.
inputs, targets = get_placeholders(input_size,output_size)dataset = tf.data.Dataset.from_tensor_slices((inputs, targets))dataset = dataset.batch(batch_size)iterator = dataset.make_initializable_iterator()x_batch, y_batch = iterator.get_next()  logits = build_network(x_batch, y_batch, num_layers, activation_fn) loss, train_op = compute_loss(logits, y_batch)accuracy = compute_accuracy(logits, y_batch)
Optimization operation
If you recollect, we hadn’t added the minimize operation to the optimizer as we wanted the values of the gradients. We can compute the gradients in Tensorflow using tf.compute_gradients. We can update the weights and biases using the above gradients using tf.apply_gradients.
grads_and_vars_tensor = train_op.compute_gradients(loss, tf.trainable_variables())optimizer = train_op.apply_gradients(grads_and_vars_tensor)
Training the model
with tf.Session() as sess:    sess.run(init_op)    def_graph = tf.get_default_graph()    gradient_means = []    losses = []    accuracies = []    train_samples = features.shape[0]        iteration = 0    trainable_variables = [var.name for var in tf.trainable_variables()]    for epoch in range(epochs):      epoch_loss = 0      total_accuracy = 0      sess.run(iterator.initializer, feed_dict={inputs:features, targets:labels})      try:                while True:          gradients = []          batch_loss, grads_and_vars,  _ , batch_accuracy = sess.run([loss, grads_and_vars_tensor, optimizer, accuracy])          var_grads = dict(zip(trainable_variables, [grad for grad,var in grads_and_vars]))          weights_grads = [var_grads[var] for var in get_weights_name(num_layers)]          mean_values = [np.mean(np.abs(val)) for val in weights_grads]          epoch_loss += batch_loss          total_accuracy += batch_accuracy*batch_size          iteration += 1          gradient_means.append(mean_values)      except tf.errors.OutOfRangeError:        pass      print("Total Iterations {}".format(iteration))      acc = total_accuracy/train_samples      accuracies.append(acc)      losses.append(epoch_loss)      print("Epoch: {}/{} , Loss: {} , Accuracy: {}".format(epoch, epochs, epoch_loss, acc))        return losses, accuracies, gradient_means
We have computed the mean of the gradients for each layer at every iteration using the below code.
var_grads = dict(zip(trainable_variables, [grad for grad,var in grads_and_vars]))weights_grads = [var_grads[var] for var in get_weights_name(num_layers)]mean_values = [np.mean(np.abs(val)) for val in weights_grads
We first create a dictionary of all the trainable variables along with the gradients. Next, we extract the gradients only for weights and compute the mean for each layer.
We have defined all the routines we need to run the model. Let’s define the model’s hyperparameters and run the model for both the sigmoid and RELU activation functions.
features = x_trainlabels = y_trainepochs = 20batch_size = 256 learning_rate = 0.001num_layers = 5hidden_units = [30,30,30,30]input_units = x_train.shape[1]output_units = y_train.shape[1] # run the model for the activation functions sigmoid and reluactivation_fns = {"sigmoid":tf.nn.sigmoid,                 "relu":tf.nn.relu}loss = {}acc= {}grad={} for name, activation_fn in activation_fns.items():  model_name = "Running model with activation function as {}".format(name)  print(model_name)  losses, accuracies, grad_means = train_model(features = features,              labels = labels,               hidden_units = hidden_units,                                               epochs = epochs,               batch_size = batch_size,               learning_rate = learning_rate,               num_layers = num_layers,               activation_fn = activation_fn)  loss[name] = losses  acc[name] = accuracies  grad[name] = grad_means
Let’s compare the accuracy and the gradients for both the models using some visualizations.
Accuracy
def plot_accuracy(accuracies, title):  for name, values in accuracies.items():    plt.plot(values, label = name)    plt.legend(title=title)plot_accuracy(acc, "Accuracy")
We can see that the accuracy of the model trained with the RELU function is more than the model trained with the sigmoid. It can also be observed that it takes more time for the sigmoid model to achieve high accuracy. WHY? Learning is slow in earlier layers due to the problem of vanishing gradients.
Let’s plot the gradients of each layer.
def plot_gradients(grads):  sigmoid = np.array(grads['sigmoid'])  relu = np.array(grads['relu'])  for layer_num in range(num_layers):    plt.figure(figsize=(20,20))    plt.subplot(5,1,layer_num+1)    plt.plot(sigmoid[:,layer_num], label='Sigmoid')    plt.plot(relu[:,layer_num], label='Relu')    plt.legend(title='Layer{}'.format(layer_num+1))plot_gradients(grad)
We can draw two major conclusions from the above graphs
The magnitude of gradients in earlier layers are very low compared to the gradients in the final layers. This explains the problem of vanishing gradients.The value of gradients of the sigmoid model is very low as compared to the RELU model.
The magnitude of gradients in earlier layers are very low compared to the gradients in the final layers. This explains the problem of vanishing gradients.
The value of gradients of the sigmoid model is very low as compared to the RELU model.
In this blog, we have discussed the problem of vanishing gradients and how it is alleviated when we use the sigmoid function. We also discussed how the RELU function helps in reducing this problem considerably.
Thanks for reading the blog. Hope you liked the article. Do leave your suggestions in the below comments.
The entire code for this article can be found in this Jupyter Notebook.