Title: Understanding weight initialization for neural networks - PyImageSearch

URL Source: https://pyimagesearch.com/2021/05/06/understanding-weight-initialization-for-neural-networks/

Published Time: 2021-05-06T05:24:00+00:00

Markdown Content:
In this tutorial, we will discuss the concept of _weight initialization_, or more simply, how we initialize our weight matrices and bias vectors.

![Image 1: weight initialization for neural networks](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20393'%3E%3C/svg%3E)

This tutorial is not meant to be a comprehensive initialization technique; however, it does highlight popular methods, but from neural network literature and general rules-of-thumb. To illustrate how these weight initialization methods work I have included basic Python/NumPy-like pseudocode when appropriate.

### **Constant Initialization**

When applying constant initialization, all weights in the neural network are initialized with a constant value, _C_. Typically _C_ will equal zero or one.

To visualize this in pseudocode let’s consider an arbitrary layer of a neural network that has 64 inputs and 32 outputs (excluding any biases for notional convenience). To initialize these weights via NumPy and zero initialization (the default used by Caffe, a popular deep learning framework) we would execute:

\>>> W = np.zeros((64, 32))

Similarly, one initialization can be accomplished via:

\>>> W = np.ones((64, 32))

We can apply constant initialization using an arbitrary of _C_ using:

\>>> W = np.ones((64, 32)) \* C

Although constant initialization is easy to grasp and understand, the problem with using this method is that it’s near impossible for us to break the symmetry of activations ([**Heinrich, 2015**](https://github.com/NVIDIA/DIGITS/blob/master/examples/weight-init/README.md)). Therefore, it is rarely used as a neural network weight initializer.

### **Uniform and Normal Distributions**

A _uniform distribution_ draws a random value from the range `[lower, upper]` where every value inside this range has _equal probability_ of being drawn.

Again, let’s presume that for a given layer in a neural network we have 64 inputs and 32 outputs. We then wish to initialize our weights in the range `lower=-0.05` and `upper=0.05`. Applying the following Python + NumPy code will allow us to achieve the desired normalization:

\>>> W = np.random.uniform(low=-0.05, high=0.05, size=(64, 32))

Executing the code above NumPy will randomly generate _64×32 = 2,048_ values from the range _\[−0.05, 0.05\],_ where each value in this range has equal probability.

We then have a _normal distribution_ where we define the probability density for the Gaussian distribution as:

**(1)** ![Image 2: p(x) = \displaystyle\frac{1}{\displaystyle\sqrt{2\pi\sigma^{2}}}e^{-\frac{(x - \mu)^{2}}{2\sigma^{2}}}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20157%2039'%3E%3C/svg%3E)

The most important parameters here are _µ_ (the mean) and _σ_ (the standard deviation). The square of the standard deviation, _σ_2, is called the variance.

When using the Keras library the `RandomNormal` class draws random values from a normal distribution with _µ = 0_ and _σ = 0.05_. We can mimic this behavior using NumPy below:

\>>> W = np.random.normal(0.0, 0.05, size=(64, 32))

Both uniform and normal distributions can be used to initialize the weights in neural networks; however, we normally impose various heuristics to create “better” initialization schemes (as we’ll discuss in the remaining sections).

### **LeCun Uniform and Normal**

If you have ever used the Torch7 or PyTorch frameworks you may notice that the default weight initialization method is called “Efficient Backprop,” which is derived by the work of [LeCun et al. (1998)](http://dl.acm.org/citation.cfm?id=645754.668382).

Here, the authors define a parameter _Fin_ (called “fan in,” or the number of _inputs_ to the layer) along with _Fout_ (the “fan out,” or number of _outputs_ from the layer). Using these values we can apply uniform initialization by:

\>>> F\_in = 64
>>> F\_out = 32
>>> limit = np.sqrt(3 / float(F\_in))
>>> W = np.random.uniform(low=-limit, high=limit, size=(F\_in, F\_out))

We can also use a normal distribution as well. The Keras library uses a truncated normal distribution when constructing the lower and upper limits, along with a zero mean:

\>>> F\_in = 64
>>> F\_out = 32
>>> limit = np.sqrt(1 / float(F\_in))
>>> W = np.random.normal(0.0, limit, size=(F\_in, F\_out))

### **Glorot/Xavier Uniform and Normal**

The default weight initialization method used in the Keras library is called “Glorot initialization” or “Xavier initialization” named after Xavier Glorot, the first author of the paper, [_Understanding the difficulty of training deep feedforward neural networks_](http://proceedings.mlr.press/v9/glorot10a.html).

For the normal distribution the `limit` value is constructed by _averaging_ the _Fin_ and _Fout_ together and then taking the square-root ([Jones, 2016](https://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization)). A zero-center (_µ_ \= 0) is then used:

\>>> F\_in = 64
>>> F\_out = 32
>>> limit = np.sqrt(2 / float(F\_in + F\_out))
>>> W = np.random.normal(0.0, limit, size=(F\_in, F\_out))

Glorot/Xavier initialization can also be done with a uniform distribution where we place stronger restrictions on `limit`:

\>>> F\_in = 64
>>> F\_out = 32
>>> limit = np.sqrt(6 / float(F\_in + F\_out))
>>> W = np.random.uniform(low=-limit, high=limit, size=(F\_in, F\_out))

Learning tends to be quite efficient using this initialization method and I recommend it for most neural networks.

### **He et al./Kaiming/MSRA Uniform and Normal**

Often referred to as “He et al. initialization,” “Kaiming initialization,” or simply “MSRA initialization,” this technique is named after Kaiming He, the first author of the paper, [_Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification_](http://arxiv.org/abs/1502.01852).

We typically use this method when we are training _very deep_ neural networks that use a ReLU-like activation function (in particular, a “PReLU,” or Parametric Rectified Linear Unit).

To initialize the weights in a layer using He et al. initialization with a _uniform distribution_ we set `limit` to be ![Image 3: limit = \sqrt{6 / F_{in}}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20114%2021'%3E%3C/svg%3E), where _F__in_ is the number of input units in the layer:

\>>> F\_in = 64
>>> F\_out = 32
>>> limit = np.sqrt(6 / float(F\_in))
>>> W = np.random.uniform(low=-limit, high=limit, size=(F\_in, F\_out))

We can also use a _normal distribution_ as well by setting _µ_ \= 0 and ![Image 4: \sigma = \sqrt{2/ F_{in}}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2087%2021'%3E%3C/svg%3E)

\>>> F\_in = 64
>>> F\_out = 32
>>> limit = np.sqrt(2 / float(F\_in))
>>> W = np.random.normal(0.0, limit, size=(F\_in, F\_out))

### **Differences in Initialization Implementation**

The actual `limit` values may vary for LeCun Uniform/Normal, Xavier Uniform/Normal, and He et al. Uniform/Normal. For example, when using Xavier Uniform in Caffe, `limit = np.sqrt(3/n)` ([Heinrich, 2015](https://github.com/NVIDIA/DIGITS/blob/master/examples/weight-init/README.md)), where _n_ is either the _Fin_, _Fout_, or their average.

On the other hand, the default Xaiver initialization for Keras uses `np.sqrt(6/(F_in + F_out))` ([Keras contributors, 2016](https://keras.io/initializers/#glorot_uniform)). No method is “more correct” than the other, but you should read the documentation of your respective deep learning library.

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

**Summary**
-----------

In this tutorial, we reviewed the fundamentals of neural networks. Specifically, we focused on the history of neural networks and the relation to biology.

From there, we moved on to _artificial neural networks_, such as the Perceptron algorithm. While important from a historical standpoint, the Perceptron algorithm has one major flaw — it cannot accurately classify nonlinear separable points. In order to work with more challenging datasets we need both (1) nonlinear activation functions and (2) multi-layer networks.

To train multi-layer networks we must use the backpropagation algorithm. We then implemented backpropagation by hand and demonstrated that when used to train multi-layer networks with nonlinear activation functions, we can model nonlinearly separable datasets, such as XOR.

Of course, implementing backpropagation by hand is an arduous process prone to bugs — we, therefore, often rely on existing libraries such as Keras, Theano, TensorFlow, etc. This enables us to focus on the actual _architecture_ rather than the underlying algorithm used to train the network.

Finally, we reviewed the four key ingredients when working with _any_ neural network, including the _dataset_, _loss function_, _model/architecture_, and _optimization method_.

Unfortunately, as some of our results demonstrated (e.g., CIFAR-10) standard neural networks fail to obtain high classification accuracy when working with challenging image datasets that exhibit variations in translation, rotation, viewpoint, etc. In order to obtain reasonable accuracy on these datasets, we’ll need to work with a special type of feedforward neural networks called _Convolutional Neural Networks_ (CNNs), which we will cover in a separate tutorial.

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)