Batch normalization in 3 levels of understanding
By: Johann Huber
Tags: Data Science, Getting Started, Deep Learning, Neural Networks, Batch Normalization

C.3) Why does Batch Normalization work ?

In most cases, Batch Normalization improves the performances of deep learning models. That’s great. But we want to know what’s actually happening inside the black box.

This is where things get a bit hairy.

The problem is : we don’t exactly know what makes Batch Normalization work so well yet. A few hypotheses are often discussed within the DL community: we will explore them step by step.

Before diving into the discussion, here are what we’ll see :

The original paper [1] assumes that BN effectiveness is due to the reduction of what they call the internal covariate shift (ICS). A recent paper [2] refutes this hypothesis. (see C.3.1)

of what they call the (ICS). A recent paper [2] refutes this hypothesis. (see C.3.1) Another hypothesis had replaced the first one with much more caution : BN mitigates the interdependency between layers during training . (see C.3.2)

. (see C.3.2) A recent paper from the MIT [2] stresses the impact of BN on the optimization landscape smoothness, making the training easier. (see C.3.3)

I bet that exploring those hypotheses will help you to build a strong intuition about Batch Normalization.

Let’s go !

C.3.1) Hypothesis n°1 — BN reduces the internal covariance shift (ICS)

Despite its fundamental impact on DNN performances, Batch Normalization is still subject to misunderstanding.

Confusions about BN are mostly due to a wrong hypothesis supported by the original article [1].

Sergey Ioffe & Christian Szegedy introduced BN as follow :

“We refer to the change in the distributions of internal nodes of a deep network, in the course of training, as Internal Covariate Shift. […] We propose a new mechanism, which we call Batch Normalization, that takes a step towards reducing internal covariate shift, and in doing so dramatically accelerates the training of deep neural nets.” — Sergey Ioffe & Christian Szegedy (source : [1])

In other words, BN is efficient because it - partially - tackles the internal Covariate Shift issue.

This statement has been severely challenged by later works [2].

Notation : From now, ICS refers to internal Covariate Shift.

To understand what causes such a confusion, let’s begin by discussing what covariate shift is, and how is it impacted by normalization.

What is covariate shift (distributional stability perspective) ?

[1]’s authors defined it clearly : covariate shift - in the distributional stability prospective - describes the shifting of a model input distribution. By extension, the internal covariate shift describes this phenomenon when it happens in the hidden layers of a deep neural network.

Let’s see why it could be an issue through an example.

Let’s assume we want to train a classifier which could answers the following question : Does this image contain a car ? If we wanted to extract all the car images from a very large unlabeled dataset, such a model would save us a huge amount of time.

We would have an RGB image as input, some convolutional layers, followed by fully connected layers. The output should be a single value, fed into a logistic function to make the final value lies between 0 and 1 - describing the probability for the input image to contain a car.

Diagram 5 : A simple CNN classifier. | Credit : author - Design : Lou HD

To train such a model, we would need a significant amount of labeled images.

Now, assuming we only have “common” cars for training. How would the model react if we ask it to classify a formula 1 car ?

As stated before, the covariate shift can make the network activations diverge (section C.2.4). Even if it doesn’t, it would deteriorate overall performances ! | Credit : Dhiva Krishna (left) and Ferhat Deniz Fors (right) on Unsplash

In this example, there is a shift between the training and testing distribution. More broadly, a different car orientation, lightning, or whether condition would be enough to impact our model performances. Here, our model does not generalize well.

If we had plot the extracted features in the features space, we would have something like this :

Diagram 6.a : Why do we need to normalize the model input value ? Case without normalization. During training, input values are scattered : the approximated function will be very accurate where there’s a high density of points. On the contrary, it will be inaccurate and submitted to randomness where the density of points is low. (For example, the approximated curve could be one of the 3 lines drawn.) | Credit : author - Design : Lou HD

Let’s assume the crosses describe features associated to images which do not contain any car, while rings describe images containing a car. Here, a single function could split the two ensembles. But the function is likely to be less accurate on the top-right part of the diagram : there are not enough points to determine a good function. This might lead the classifier to do many errors during evaluation.

To efficiently train our model, we would require many car images, in any possible context we could imagine. Even if this is still how we train our CNN nowadays, we want to make sure our model will generalize well using as few examples as possible.

The problem could be summarized as follow :

From the model point-of-view, training images are - statistically - too different from testing images.

There is a covariate shift.

We can face this issue using simpler models. It is well known that linear regression models are easier to optimize when the input values are normalized (i.e. making its distribution close to (𝜇 = 0, σ = 1)) : that’s why we usually normalize the input values of a model.

Diagram 6.b : Why do we need to normalize the model input value ? Case with normalization. Normalizing the input signal makes the points closer to each other in the feature space during training : it is now easier to find a well generalizing function. | Credit : author — Design : Lou HD

This solution was already well known before the publication of the BN paper. With BN, [1]’s authors wanted to extend this method to the hidden layers to help training.

Internal covariate shift deteriorates training : the original paper hypothesis

Diagram 7 : Internal covariate shift (ICS) principle in the distributional stability perspective (ICS_distrib). | Credit : author - Design : Lou HD

In our car classifier, we can see hidden layers as units which are activated when they identify some “conceptual” features associated with cars : it could be a wheel, a tire or a door. We can suppose that the previously described effect could happen inside hidden units. A wheel with a certain orientation angle will activate neurons with a specific distribution. Ideally, we want to make some neurons react with a comparable distribution for any wheel orientations, so the model could efficiently conclude on the probability for the input image to contain a car.

If there is a huge covariate shift in the input signal, the optimizer will have trouble generalizing well. On the contrary, if the input signal always follows a standard normal distribution, the optimizer will more easily generalize. With this in mind, [1]’s authors applied the strategy of normalizing the signal in the hidden layers. They assumed that forcing to (𝜇 = 0, σ = 1) the intermediates signal distribution will help the network generalization in the “conceptual” levels of features.

Though, we do not always want standard normal distribution in the hidden units. It would reduce the model representativity :

Diagram 8 : Why don’t we always want a standard normal distribution in the hidden units. Here, the sigmoid function only works in its linear regime. | Credit : author - Design : Lou HD

The original article takes the sigmoid function as an example to show why normalization alone is an issue. If the input signal values lie between 0 and 1, the nonlinear function would only work … in its linear regime. Sounds problematic.

To tackle this issue, they added two trainable parameters 𝛽 and 𝛾, allowing the optimizer to define the optimum mean (using 𝛽) and standard deviation (using 𝛾) for a specific task.

⚠Warning : The following hypothesis is now outdated. A lot of great content about BN still claim it as the reason that makes the method works in practice. Yet, recent works severely challenged it.

For a couple of years after the release of [1], the DL community explained BN effectiveness as follow :

— Hypothesis 1 — — — — — — — — — — — — — — — — — — — — — BN ➜ Normalization of the input signal inside hidden units ➜ Adding two trainable parameters to adjust the distribution and get the most out of the nonlinearities ➜ Easier training

Here, normalization to (𝜇 = 0, σ = 1) is what mostly explains BN’s effectiveness. This hypothesis has been challenged (see section C.3.3), replaced by another hypothesis :

— Hypothesis 2 — — — — — — — — — — — — — — — — — — — — — BN ➜ Normalization of the input signal inside hidden units ➜ Reduces interdependency between hidden layers (in a distribution stability perspective) ➜ Easier training

There’s is a slight but very important difference. Here, the purpose of normalization is to reduce interdependency between layers (in a distribution stability perspective), so the optimizer could choose the optimum distribution by adjusting only two parameters ! Let’s explore this hypothesis a little bit further.

What’s the link between Batch Normalization and this picture ? | Credit : Danilo Alvesd on Unsplash

C.3.2) Hypothesis n°2 — BN mitigates interdependency between hidden layers during training

About this section : I could not find any solid evidence about the hypothesis discussed in this section. Therefore, I decided to mostly rely on Ian Goodfellow’s explanations on that matter (in particular in this brilliant video).

Consider the following example :

Diagram 9 : A simple DNN, which consists of linear transformations. | Inspired by Ian Goodfellow

Where (a), (b), (c), (d) and (e) are the sequential layers of the network. Here is a very simple example, where all layers are linked by linear transformations. Let’s assume we want to train such model using SGD.

To update the weights of the layer (a), we need to calculate the gradient from the network’s output :

We first consider a network without BN layers. From the above equation, we conclude that if all gradients are large, grad(a) will be very large. On the contrary, if all gradients are small, grad(a) will be almost negligible.

It is pretty easy to see how dependent are layers from each other by looking at the input distribution of hidden units : a modification of (a) weights will modify the input distribution of (b) weights, which will eventually modify the input signal of (d) and (e). This interdependency could be problematic for training stability : if we want to adjust the input distribution of a specific hidden unit, we need to consider the whole sequence of layers.

However, SGD considers 1st order relationships between layers. So they don’t take into account the higher degree relationships mentioned above !

Diagram 10 : Hypothesis n°2 principle. BN layers make the signal regulation easier, by normalizing the signal inside each hidden unit, and allowing distribution adjustment using 𝛽 and 𝛾. BN acts like a valve which makes the flow control easier at some points, without deteriorating the underlying complexity of the network ! | Credit : author - Design : Lou HD

Adding BN layers significantly reduces the interdependence between each layers (in the distribution stability perspective) during training. Batch Normalization acts like a valve which holds back the flow, and allows its regulation using 𝛽 et 𝛾. It is then no longer necessary to take all parameters into account to have clues about distribution inside the hidden units.

Remark : The optimizer can then do larger weights modifications without deteriorating adjusted parameters in other hidden layers. It makes the hyperparameter tuning way easier !

This example puts aside the hypothesis which claims that BN effectiveness is due to the normalization of intermidiates signal distribution to (𝜇 = 0, σ = 1).

Here, BN aims to make the optimizer job easier, allowing it to adjust hidden layer distribution with only two parameters at a time.

⚠ However, keep in mind that this is mostly speculation. Those discussions should be used as insights to build intuition about BN. We still don’t exactly know why BN is efficient in practice !

In 2019, a team of researchers from MIT carried out some interesting experiments about BN [2]. Their results severely challenge the hypothesis n°1 (still shared by many serious blog posts and MOOCS !).

We should have a look on this paper if we want to avoid “local minima hypotheses” about the BN impact on training… ;)

Alright … you better initialize well. | Credit : Tracy Zhang on Unsplash

C.3.3) Hypothesis n°3 — BN makes the optimisation landscape smoother

About this section : I’ve synthesized results from [2] which could help us to build a better intuition about BN. I wasnt able to be exhaustive, this paper is dense, I recommend you to read it thoroughly if you’re interested in those concepts.

Let’s jump straight into the 2nd experiment of [2]. Their goal was to check the correlation between ICS, and the benefits of BN on training performances (hypothesis n°1).

Notation : We’ll now refer to this covariate shift by ICS_distrib.

To do so, the researchers have trained three VGG networks (on CIFAR-10) :

The first one does not have any BN layer ;

The second one does have BN layers ;

The third one is similar to the second one, except from the fact that they have explicitly added some ICS_distrib inside the hidden unit right before the activation (by adding random bias & variance).

They measured the accuracy reached by each model, and the evolution of the distribution values w.r.t iterations. Here is what they got :

Diagram 6 : BN on ICS_distrib | Networks with BN are trained faster than the standard one ; explicitly adding ICS_distrib on a regulated network does not deteriorate BN benefits. | Source : [2]

We can see that the 3rd network has, as expected, a very high ICS. However, the noisy network is still trained faster than the standard one. Its reached performances are comparable to the ones obtained with a standard BN network. This result suggests that the BN effectiveness is not related to ICS_distrib. Oops !

We shouldn’t discard the ICS theory too fast : if the BN effectiveness does not come from ICS_distrib, it might be related to another definition of ICS. After all, the intuition behind the hypothesis n°1 makes sense, doesn’t it ?

The main issue with ICS_distrib is that its definition is related to input distribution of hidden units. So there is no direct link with the optimization problem on its own.

[2]’s authors proposed another definition of ICS :

Let’s consider a fixed input X.

We define the internal covariate shift from an optimization perspective as the difference between the gradient computed on a hidden layer k after backpropagating the error L(X)_it, and the gradient computed on the same layer k from the loss L(X)_it+1 computed after the iteration = it update of weights.

This definition aims to focus on the gradients more than on the hidden layer input distributions, assuming that it could give us better clues on how ICS could have an impact on the underlying optimization problem.

Notation: ICS_opti now refers to the ICS defined from an optimization perspective.

In the next experiment, authors evaluate ICS_opti impact on training performances. To do so, they measure the variation of ICS_opti during training for a DNN with and without BN layers. To quantify the variation of gradient mentioned in the ICS_opti definition, they calculate :

L2 difference : Do the gradients have a close norm before and after the weights update ? Ideally : 0

Do the gradients have a close norm before and after the weights update ? Ideally : 0 Cosine angle : Do the gradients have a close orientation before and after the weights update ? Ideally : 1

Figure 7 : BN impact on ICS_opti | L2 diff and cosine angles suggest that BN does not prevent ICS_opti (it seems to slightly increase it, somehow). | Source [2]

Results are a bit surprising : the network which relies on BN seems to have a higher ICS_opti than the standard network. Remember that the network with BN (blue curve) is trained faster than the standard network (red curve) !

ICS seems definitively not related to training performances…at least for the explored definition of ICS.

Somehow, Batch Normalization have another impact on the network, which makes convergence easier.

Now, let’s study how does BN affects the optimization landscape. We may find clues there.

Here is the last experiment covered in this story :

Figure 11 : Optimization landscape exploration in the gradient’s direction. Experiment carried out in the paper [2] | Inspired by Andrew Ilyas - Design : Lou HD

From a single gradient, we update weights with different optimization steps (which act like a learning rate). Intuitively, we define a direction from a certain point (i.e. a network configuration ω) in the feature space, and then explore the optimization landscape further and further in this direction.

At each step, we measure the gradient and the loss. We can therefore compare different points of the optimization landscape with a starting point. If we measure large variations, the landscape is very unstable and the gradient is incertain : big steps might deteriorate optimization. On the contrary, if the measured variations are small, the landscape is stable and the gradient is trustworthy : we can apply larger steps without compromising optimization. In other words, we can use a larger learning rate, and make the convergence faster (a well known properties of BN).

Let’s have a look on the results :

Figure 8 : BN impact on optimization landscape smoothing | Using BN significantly reduces gradient variations. | Source : [2]

We can clearly see that the optimization landscape is way smoother with BN layers.

We finally have results we can rely on to explain BN effectiveness : BN layer makes somehow the optimization landscape smoother. That makes the optimizer job easier : we can define a larger learning rate without being submitted to gradient vanishing (weights stuck on sudden flat surfaces) or to gradient explosion (weights fallen in an abrupt local minima).

We can know formulate a 3rd hypothesis, proposed by [2] :

— Hypothesis 3 : — — — — — — — — — — — — — — — — — — BN ➜ Normalization of the input signal inside hidden units ➜ Makes the optimization landscape smoother ➜ Faster and more stable training

It raises another question : How does BN make the optimization landscape smoother ?

[2]’s authors also explored those matters from a theoretical point of view. Their work is very instructive, helping to get a better grasp of the smoothing effect of Batch Normalization. In particular, they showed that BN makes the optimization landscape smoother while preserving all of the minima of the normal landscape. In other words, BN reparametrizes the underlying optimization problem, making the training faster and easier !

⚠ In additional studies, [2]’s authors observed that this effect is not unique to BN. They obtained similar training performances with other normalization methods (for example L1 or L2 normalization). Those observations suggest that BN effectiveness is mostly due to serendipity, leveraging on underlying mechanisms that we have not perfectly identified yet.

Now it’s time to set a very high learning rate. | Credit : Finding Dan | Dan Grinwis on Unsplash

To conclude this section, this paper severely challenges the widely admitted idea that BN effectiveness is mostly due to ICS reduction (in a training stability distribution perspective, as well as in an optimization perspective). However, it stresses the impact of BN smoothing effect on the optimization landscape.

While this paper states a hypothesis about BN impact on training speed, it does not answer why BN helps generalization.

They briefly argue that making the optimization landscape smoother could help the model to converge on flat minima, which have better generalizing properties. No more details on that matter, though.

Their main contribution is to challenge the commonly admitted idea of BN effect on ICS — which is already significant !

C.4) Summary : Why does BN work ? What do we know so far

Hypothesis 1 : BN layer mitigates the internal covariate shift (ICS)

❌ Wrong : [2] showed that in practice, there is no correlation between ICS and training performances.

Hypothesis 2 : BN layer makes the optimizer job easier by allowing it to adjust input distribution of hidden units with only 2 parameters.

❓ Maybe : This hypothesis highlights the interdependency between parameters, making the optimization task harder. No solid proof, though.

Hypothesis 3 : BN layer reparametrize the underlying optimization problem, making it smoother and more stable.

❓ Maybe : Their results are quite recent. To my knowledge, they have not been challenged so far. They provide empirical demonstrations and pieces of theoretical justifications, but some fundamental questions remain unanswered (such as “how does BN help generalization ?”).

Discussion : It seems to me that the last two hypotheses are compatible. Intuitively, we could see the hypothesis n°2 as a projection from a problem with many parameters, to many problems with a couple of parameters ; a kind of dimensionality reduction, which would helps generalization. Any ideas about it ?

Many questions remain open, and Batch Normalization is still a topic of research nowadays. Discussing those hypotheses still helps to get a better understanding of this commonly used method, discarding some erroneous statements we had in mind for a couple of years.

Those questions do not prevent us to leverage on the benefits of BN in practice, though !

Conclusion

Batch Normalization (BN) is one of the most important advances in the field of Deep Learning (DL) in recent years. Relying on two successive linear transformations, this method makes Deep Neural network (DNN) training faster and more stable.

The most widely admitted hypothesis about what makes BN efficient in practice is the reduction of interdependence between hidden layers during training. However, the normalization transformation impact of optimization landscape smoothness seems to be an important mechanism of BN effectiveness.

Many commonly used DNN rely on BN nowadays (ex : ResNet [4], EfficientNet [5], …).

If you are interested in Deep Learning, you will for sure have to get familiar with this method !

Opened questions

Even if BN appears to be efficient in practice for years, many questions about its underlying mechanisms remain unanswered.

Here is a non-exhaustive list of opened questions about BN :

How does BN helps generalization ?

? Is BN the best normalization method to make the optimization easier ?

to make the optimization easier ? How do 𝛽 et 𝛾 impact the optimization landscape smoothness ?

impact the optimization landscape smoothness ? Experiments carried out by [2] to explore optimization landscape focus on short-term impact of BN on gradient : they have measured the variation of gradient & loss from a single iteration, for several values of step. How does BN impact gradient on the long run ? Do weights interdependency have other interesting impact on the optimization landscape ?

Acknowledgment

Many thanks to Lou Hacquet-Delepine for all the drawn diagrams, as well as her overall help in proofreading !

References

[1] Ioffe, S., & Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift, arXiv preprint arXiv:1502.03167.

[2] Santurkar, S., Tsipras, D., Ilyas, A., & Madry, A. (2018). How does batch normalization help optimization?, Advances in Neural Information Processing Systems

[3] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., … & Rabinovich, A. (2015). Going deeper with convolutions, Proceedings of the IEEE conference on computer vision and pattern recognition

[4] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition

[5] Tan, M., & Le, Q. V. (2019). Efficientnet: Rethinking model scaling for convolutional neural networks, arXiv preprint arXiv:1905.11946.

[6] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. Bengio, Y. (2014), Generative adversarial nets, Advances in neural information processing systems

Going further