Dropout Neural Network Layer In Keras Explained | by Cory Maklin | Towards Data Science

Machine learning is ultimately used to predict outcomes given a set of features. Therefore, anything we can do to generalize the performance of our model is seen as a net gain. Dropout is a technique used to prevent a model from overfitting. Dropout works by randomly setting the outgoing edges of hidden units (neurons that make up hidden layers) to 0 at each update of the training phase. If you take a look at the Keras documentation for the dropout layer, you’ll see a link to a white paper written by Geoffrey Hinton and friends, which goes into the theory behind dropout.
Journal of Machine Learning Research 15 (2014) 1929-1958 Submitted 11/13; Published 6/14

Dropout: A Simple Way to Prevent Neural Networks from

Overfitting

Nitish Srivastava nitish@cs.toronto.edu

Geoffrey Hinton hinton@cs.toronto.edu

Alex Krizhevsky kriz@cs.toronto.edu

Ilya Sutskever ilya@cs.toronto.edu

Ruslan Salakhutdinov rsalakhu@cs.toronto.edu

Department of Computer Science

University of Toronto

10 Kings College Road, Rm 3302

Toronto, Ontario, M5S 3G4, Canada.

Editor: Yoshua Bengio

Abstract

Deep neural nets with a large number of parameters are very powerful machine learning

systems. However, overfitting is a serious problem in such networks. Large networks are also

slow to use, making it difficult to deal with overfitting by combining the predictions of many

different large neural nets at test time. Dropout is a technique for addressing this problem.

The key idea is to randomly drop units (along with their connections) from the neural

network during training. This prevents units from co-adapting too much. During training,

dropout samples from an exponential number of different “thinned” networks. At test time,

it is easy to approximate the effect of averaging the predictions of all these thinned networks

by simply using a single unthinned network that has smaller weights. This significantly

reduces overfitting and gives major improvements over other regularization methods. We

show that dropout improves the performance of neural networks on supervised learning

tasks in vision, speech recognition, document classification and computational biology,

obtaining state-of-the-art results on many benchmark data sets.

Keywords: neural networks, regularization, model combination, deep learning

1. Introduction

Deep neural networks contain multiple non-linear hidden layers and this makes them very

expressive models that can learn very complicated relationships between their inputs and

outputs. With limited training data, however, many of these complicated relationships

will be the result of sampling noise, so they will exist in the training set but not in real

test data even if it is drawn from the same distribution. This leads to overfitting and many

methods have been developed for reducing it. These include stopping the training as soon as

performance on a validation set starts to get worse, introducing weight penalties of various

kinds such as L1 and L2 regularization and soft weight sharing (Nowlan and Hinton, 1992).

With unlimited computation, the best way to “regularize” a fixed-sized model is to

average the predictions of all possible settings of the parameters, weighting each setting by

c 2014 Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever and Ruslan Salakhutdinov.

Srivastava, Hinton, Krizhevsky, Sutskever and Salakhutdinov

(a) Standard Neural Net (b) After applying dropout.

Figure 1: Dropout Neural Net Model. Left: A standard neural net with 2 hidden layers. Right:

An example of a thinned net produced by applying dropout to the network on the left.

Crossed units have been dropped.

its posterior probability given the training data. This can sometimes be approximated quite

well for simple or small models (Xiong et al., 2011; Salakhutdinov and Mnih, 2008), but we

would like to approach the performance of the Bayesian gold standard using considerably

less computation. We propose to do this by approximating an equally weighted geometric

mean of the predictions of an exponential number of learned models that share parameters.

Model combination nearly always improves the performance of machine learning meth-
ods. With large neural networks, however, the obvious idea of averaging the outputs of

many separately trained nets is prohibitively expensive. Combining several models is most

helpful when the individual models are different from each other and in order to make

neural net models different, they should either have different architectures or be trained

on different data. Training many different architectures is hard because finding optimal

hyperparameters for each architecture is a daunting task and training each large network

requires a lot of computation. Moreover, large networks normally require large amounts of

training data and there may not be enough data available to train different networks on

different subsets of the data. Even if one was able to train many different large networks,

using them all at test time is infeasible in applications where it is important to respond

quickly.

Dropout is a technique that addresses both these issues. It prevents overfitting and

provides a way of approximately combining exponentially many different neural network

architectures efficiently. The term “dropout” refers to dropping out units (hidden and

visible) in a neural network. By dropping a unit out, we mean temporarily removing it from

the network, along with all its incoming and outgoing connections, as shown in Figure 1.

The choice of which units to drop is random. In the simplest case, each unit is retained with

a fixed probability p independent of other units, where p can be chosen using a validation

set or can simply be set at 0.5, which seems to be close to optimal for a wide range of

networks and tasks. For the input units, however, the optimal probability of retention is

usually closer to 1 than to 0.5.

1930

Dropout

Present with

probability p

w

✲

(a) At training time

Always

present

pw

✲

(b) At test time

Figure 2: Left: A unit at training time that is present with probability p and is connected to units

in the next layer with weights w. Right: At test time, the unit is always present and

the weights are multiplied by p. The output at test time is same as the expected output

at training time.

Applying dropout to a neural network amounts to sampling a “thinned” network from

it. The thinned network consists of all the units that survived dropout (Figure 1b). A

neural net with n units, can be seen as a collection of 2n possible thinned neural networks.

These networks all share weights so that the total number of parameters is still O(n

2

), or

less. For each presentation of each training case, a new thinned network is sampled and

trained. So training a neural network with dropout can be seen as training a collection of 2n

thinned networks with extensive weight sharing, where each thinned network gets trained

very rarely, if at all.

At test time, it is not feasible to explicitly average the predictions from exponentially

many thinned models. However, a very simple approximate averaging method works well in

practice. The idea is to use a single neural net at test time without dropout. The weights

of this network are scaled-down versions of the trained weights. If a unit is retained with

probability p during training, the outgoing weights of that unit are multiplied by p at test

time as shown in Figure 2. This ensures that for any hidden unit the expected output (under

the distribution used to drop units at training time) is the same as the actual output at

test time. By doing this scaling, 2n networks with shared weights can be combined into

a single neural network to be used at test time. We found that training a network with

dropout and using this approximate averaging method at test time leads to significantly

lower generalization error on a wide variety of classification problems compared to training

with other regularization methods.

The idea of dropout is not limited to feed-forward neural nets. It can be more generally

applied to graphical models such as Boltzmann Machines. In this paper, we introduce

the dropout Restricted Boltzmann Machine model and compare it to standard Restricted

Boltzmann Machines (RBM). Our experiments show that dropout RBMs are better than

standard RBMs in certain respects.

This paper is structured as follows. Section 2 describes the motivation for this idea.

Section 3 describes relevant previous work. Section 4 formally describes the dropout model.

Section 5 gives an algorithm for training dropout networks. In Section 6, we present our

experimental results where we apply dropout to problems in different domains and compare

it with other forms of regularization and model combination. Section 7 analyzes the effect of

dropout on different properties of a neural network and describes how dropout interacts with

the network’s hyperparameters. Section 8 describes the Dropout RBM model. In Section 9

we explore the idea of marginalizing dropout. In Appendix A we present a practical guide

1931

Srivastava, Hinton, Krizhevsky, Sutskever and Salakhutdinov

for training dropout nets. This includes a detailed analysis of the practical considerations

involved in choosing hyperparameters when training dropout networks.

2. Motivation

A motivation for dropout comes from a theory of the role of sex in evolution (Livnat et al.,

2010). Sexual reproduction involves taking half the genes of one parent and half of the

other, adding a very small amount of random mutation, and combining them to produce an

offspring. The asexual alternative is to create an offspring with a slightly mutated copy of

the parent’s genes. It seems plausible that asexual reproduction should be a better way to

optimize individual fitness because a good set of genes that have come to work well together

can be passed on directly to the offspring. On the other hand, sexual reproduction is likely

to break up these co-adapted sets of genes, especially if these sets are large and, intuitively,

this should decrease the fitness of organisms that have already evolved complicated co-
adaptations. However, sexual reproduction is the way most advanced organisms evolved.

One possible explanation for the superiority of sexual reproduction is that, over the long

term, the criterion for natural selection may not be individual fitness but rather mix-ability

of genes. The ability of a set of genes to be able to work well with another random set of

genes makes them more robust. Since a gene cannot rely on a large set of partners to be

present at all times, it must learn to do something useful on its own or in collaboration with

a small number of other genes. According to this theory, the role of sexual reproduction

is not just to allow useful new genes to spread throughout the population, but also to

facilitate this process by reducing complex co-adaptations that would reduce the chance of

a new gene improving the fitness of an individual. Similarly, each hidden unit in a neural

network trained with dropout must learn to work with a randomly chosen sample of other

units. This should make each hidden unit more robust and drive it towards creating useful

features on its own without relying on other hidden units to correct its mistakes. However,

the hidden units within a layer will still learn to do different things from each other. One

might imagine that the net would become robust against dropout by making many copies

of each hidden unit, but this is a poor solution for exactly the same reason as replica codes

are a poor way to deal with a noisy channel.

A closely related, but slightly different motivation for dropout comes from thinking

about successful conspiracies. Ten conspiracies each involving five people is probably a

better way to create havoc than one big conspiracy that requires fifty people to all play

their parts correctly. If conditions do not change and there is plenty of time for rehearsal, a

big conspiracy can work well, but with non-stationary conditions, the smaller the conspiracy

the greater its chance of still working. Complex co-adaptations can be trained to work well

on a training set, but on novel test data they are far more likely to fail than multiple simpler

co-adaptations that achieve the same thing.

3. Related Work

Dropout can be interpreted as a way of regularizing a neural network by adding noise to

its hidden units. The idea of adding noise to the states of units has previously been used in

the context of Denoising Autoencoders (DAEs) by Vincent et al. (2008, 2010) where noise

1932

In the proceeding example, we’ll be using Keras to build a neural network with the goal of recognizing hand written digits.
from keras.datasets import mnistfrom matplotlib import pyplot as pltplt.style.use('dark_background')from keras.models import Sequentialfrom keras.layers import Dense, Flatten, Activation, Dropoutfrom keras.utils import normalize, to_categorical
We use Keras to import the data into our program. The data is already split into the training and testing sets.
(X_train, y_train), (X_test, y_test) = mnist.load_data()
Let’s have a look to see what we’re working with.
plt.imshow(x_train[0], cmap = plt.cm.binary)plt.show()
After we’re done training out model, it should be able to recognize the preceding image as a five.
There is a little preprocessing that we must perform beforehand. We normalize the pixels (features) such that they range from 0 to 1. This will enable the model to converge towards a solution that much faster. Next, we transform each of the target labels for a given sample into an array of 1s and 0s where the index of the number 1 indicates the digit the the image represents. We do this because otherwise our model would interpret the digit 9 as having a higher priority than the number 3.
X_train = normalize(X_train, axis=1)X_test = normalize(X_test, axis=1)y_train = to_categorical(y_train)y_test = to_categorical(y_test)
Before feeding a 2 dimensional matrix into a neural network, we use a flatten layer which transforms it into a 1 dimensional array by appending each subsequent row to the one that preceded it. We’re going to be using two hidden layers consisting of 128 neurons each and an output layer consisting of 10 neurons, each for one of the 10 possible digits. The softmax activation function will return the probability that a sample represents a given digit.
model = Sequential()model.add(Flatten(input_shape=(28, 28)))model.add(Dense(128))model.add(Activation('relu'))model.add(Dense(128))model.add(Activation('relu'))model.add(Dense(10))model.add(Activation('softmax'))model.summary()
Since we’re trying to predict classes, we use categorical crossentropy as our loss function. We will measure the performance of the model using accuracy.
model.compile(    loss='categorical_crossentropy',    optimizer='adam',    metrics=['accuracy'])
We set 10% of the data aside for validation. We will use this to compare the tendency of a model to overfit with and without dropout. A batch size of 32 implies that we will compute the gradient and take a step in the direction of the gradient with a magnitude equal to the learning rate, after having pass 32 samples through the neural network. We do this a total of 10 times as specified by the number of epochs.
history = model.fit(    X_train,    y_train,    epochs=10,    batch_size=32,    validation_split=0.1,    verbose = 1,    shuffle=True)
We can plot the training and validation accuracies at each epoch by using the history variable returned by the fit function.
loss = history.history['loss']val_loss = history.history['val_loss']epochs = range(1, len(loss) + 1)plt.plot(epochs, loss, 'y', label='Training loss')plt.plot(epochs, val_loss, 'r', label='Validation loss')plt.title('Training and validation loss')plt.xlabel('Epochs')plt.ylabel('Loss')plt.legend()plt.show()
As you can see, without dropout, the validation loss stops decreasing after the third epoch.
acc = history.history['acc']val_acc = history.history['val_acc']plt.plot(epochs, acc, 'y', label='Training acc')plt.plot(epochs, val_acc, 'r', label='Validation acc')plt.title('Training and validation accuracy')plt.xlabel('Epochs')plt.ylabel('Accuracy')plt.legend()plt.show()
As you can see, without dropout, the validation accuracy tends to plateau around the third epoch.
Using this simple model, we still managed to obtain an accuracy of over 97%.
test_loss, test_acc = model.evaluate(X_test, y_test)test_acc
There’s some debate as to whether the dropout should be placed before or after the activation function. As a rule of thumb, place the dropout after the activate function for all activation functions other than relu. In passing 0.5, every hidden unit (neuron) is set to 0 with a probability of 0.5. In other words, there’s a 50% change that the output of a given neuron will be forced to 0.
model_dropout = Sequential()model_dropout.add(Flatten(input_shape=(28, 28)))model_dropout.add(Dense(128))model_dropout.add(Dropout(0.5))model_dropout.add(Activation('relu'))model_dropout.add(Dense(128))model_dropout.add(Dropout(0.5))model_dropout.add(Activation('relu'))model_dropout.add(Dense(10))model_dropout.add(Activation('softmax'))model_dropout.summary()
Again, since we’re trying to predict classes, we use categorical crossentropy as our loss function.
model_dropout.compile(    loss='categorical_crossentropy',    optimizer='adam',    metrics=['accuracy'])
By providing the validations split parameter, the model will set apart a fraction of the training data and will evaluate the loss and any model metrics on this data at the end of each epoch. If the premise behind dropout holds, then we should see a notable difference in the validation accuracy compared to the previous model. The shuffle parameter will shuffle the training data before each epoch.
history_dropout = model_dropout.fit(    X_train,    y_train,    epochs=10,    batch_size=32,    validation_split=0.1,    verbose = 1,    shuffle=True)
As you can see, the validation loss is significantly lower than that obtained using the regular model.
loss = history_dropout.history['loss']val_loss = history_dropout.history['val_loss']epochs = range(1, len(loss) + 1)plt.plot(epochs, loss, 'y', label='Training loss')plt.plot(epochs, val_loss, 'r', label='Validation loss')plt.title('Training and validation loss')plt.xlabel('Epochs')plt.ylabel('Loss')plt.legend()plt.show()
As you can see, the model converged much faster and obtained an accuracy of close to 98% on the validation set, whereas the previous model plateaued around the third epoch.
acc = history_dropout.history['acc']val_acc = history_dropout.history['val_acc']plt.plot(epochs, acc, 'y', label='Training acc')plt.plot(epochs, val_acc, 'r', label='Validation acc')plt.title('Training and validation accuracy')plt.xlabel('Epochs')plt.ylabel('Accuracy')plt.legend()plt.show()
The accuracy obtained on the testing set isn’t very different than the one obtained from the model without dropout. This is in all likelihood due to the limited number of samples.
test_loss, test_acc = model_dropout.evaluate(X_test, y_test)test_acc
Dropout can help a model generalize by randomly setting the output for a given neuron to 0. In setting the output to 0, the cost function becomes more sensitive to neighbouring neurons changing the way the weights will be updated during the process of backpropagation.