Regularization Techniques — a list down
Tags: Regularization, Batch Normalization, Deep Learning, Dropout Regularization, Fastai

Regularization Techniques — a list down

Here I will theoretically discuss various regularization techniques used in the learning process.

Image by Виктория Бородинова from Pixabay

General Knowledge

Regularization techniques are used to reduce overfitting in the learning process. Overfitting occurs when our model learns the input so much better that it could only further resolve the scenarios that seem similar to the input. It behaves very badly if something gets out of the scope of the input. Overfitting causes a lot of problems if we have our overfitted model in production. Let’s learn the techniques.

One way to reduce overfitting is to reduce the number of parameters — that’s a conventional approach. Now, instead of reducing the number of parameters, we use all the parameters, but we penalize the non-required, unimportant parameters which do not affect the output. We penalize those parameters for being non-zero, for their existence — that’s an unconventional approach.

One thing you should keep in mind is that regularization does not help in optimizing the cost function, it only helps in reducing the overfitting i.e. it is used to overcome the low bias & high variance behavior of our model.

Dropouts

The first type of regularization technique is Dropout. This is an exciting type of regularization technique. As per this technique, we remove a random number of activations. You can easily guess that removing the random activations will lead to smaller networks and smaller networks are always better to train. Now, activations are the output obtained by multiplying the weights and inputs. Activations tell about the input since they were derived from the inputs only. So, if we remove a particular part of activations at each layer, therefore no specific activation would learn the input model. Thus, there would not be any specific activation on which we would put extra weight since we do not know if that activation would sustain or not. Therefore, there won’t be any overfitting regarding the input model. Now, the probability of removing any activation is decided by the machine learning practitioner.

Now, sometimes rather than only removing the activations, we sometimes also remove the inputs. This is certainly not a regular behavior to remove the input, but depending upon your model, eliminating the inputs also helps.

Image source: Google

The first layer in the image represents the inputs. Afterward, layers represent the activations to that layer.

You may see that we have removed some inputs in the first layer, as shown in the second diagram. We have also removed some of the activations in the subsequent layers. That’s how we define dropout.

Now, we can initialize a different amount of dropout value for different layers. For example, for layers having 10 hidden units, we could define the drop probability to 0.5 in comparison to a layer having only 4 hidden units for which w=our drop probability would be just 0.2. For, even small layers, we could define drop probability to 0.

We do not define dropouts for the continuous variables because defining dropouts for the continuous variables lead to vanishing their existence literally. But, we define dropout for embeddings. This is because embeddings are just lookups obtained by multiplying a one-hot encoded matrix with the input weights. Therefore, we may remove them.

In pretty much every fast.ai learner, there’s a parameter called ps , which will be the p-value for the dropout for each layer. An embedding dropout is just a dropout. We define it by emb_drop .

There is an exciting feature of dropout. We talk about training time and test time (we also call inference time). Training time is when we’re doing that those weight updates — the backpropagation. The training time, dropout works the way we just saw. At test time, we turn off dropout. We’re not going to do dropout anymore because we wanted to be as accurate as possible. We’re not training, so we can’t cause it to overfit when we’re doing inference. So we remove dropout at the test time.

Now, dropout has to be carried out in forwarding and backward propagation of the model. Let us understand why?

It is significant to account for anything that you’re doing in the forward propagation in the backward step as well — otherwise, you are computing a gradient of a function that is different from the one that you’re evaluating.

In forward propagation, inputs are set to zero with probability p , and otherwise scaled up by 1/1-p in order to keep the scaling of the activations.

, and otherwise scaled up by in order to keep the scaling of the activations. In backward propagation, gradients for the same dropped units are zeroed out; other gradients are scaled up by the same 1/1-p in order to keep the scaling of the activations.

Batch Normalization

Batch Normalization is another important regularization technique that helps a lot, preventing overfitting and training in the right order.

The intuition behind the Batch Normalization

We always normalize the inputs in the case of Logistic Regression in order to train and learn the parameters better. Now, if we think of deep neural networks, then we want to normalize a lot of weights and biases. Weights and bias in the first layer will be easier to optimize as the input to them would be normalized X. But what about weights and biases in the latter layers. So, for their optimization, we normalize the activations so as so learn the latter parameters better.

Now, the question comes after this is, What if we do not want to normalize the activations for a certain layer? To handle this case, we multiply the normalized activation input with the gamma parameter and add beta parameter and train these parameters in the traditional manner. Using gamma and beta prevents the forcefull normalizing of the activations.

Why do we need Batch Normalization?

Consider a situation where we have 2 linear layers(LL1 and LL2) and 1 non-linear layer (NL1).

LL2 = NL1(LL1)

Now activations at LL2 are highly dependent on output from the previous layers which keep on changing due to backpropagation which unstabilizes the whole neural network to much extent. Thus, we need to normalize the data in order to reduce these variations.